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(57) Abstract 

The invention 
provides a virtual assembly 
design environment (VAE) 
that can simulate axial 
and planar constrained 
motion for multiple parts 
in any combination and 
application order. Each 
sequence of assembly 
operations can be recorded 
and stiMed for later usage. 
A guidance mechanism 
may be employed by a 
user to assist in pertbrming 
the assembly operations. 
Dynamic simulation 
methods may be used to 
simulate object behavior 
in the VAE usmg physical 
laws and collision detection 
algorithms. The physical 
properties of the parts can 
be created in a separate 
CAD system (including 
mass properties). In 
the invention, physical 
property infonnation is 
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^ansfemed fmm the CAD system to a virtual reality environment where it is used in dynamic simulations. The parts behave realistically 
in the user's hand, constrained on the base part, or moving freely in space. A swept volume can be generated directly in the CAD system 
which is more accurate and compact than those created using numerical methods and can be easily pnxessed by CAD systems. A swept 
volume trajectory editing mechanism has been implemented. Real time bi-directional data transfer between the VR environment and the 
CAD system has been achieved. The user can perform parametric design modifications in the virtual environment through a CAD system. 
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METHOD AND SYSTEM FOR A VIRTUAL ASSEMBLY DESK^ 

ENVIRONMENT 

Related Patent Application 
5 This application claims priority from U.S. Provisional Application Serial No. 

60/1 13,629, filed Decemb«- 23, 1998. 

Field of the Invention 
This utility patent application relates generally to the field of virtual reality 
(VR), and more specifically, to employing a virtual reality environment integrated with 
10 a computer aided design (CAD) system to simulate the virtual assembly of a finished 
product. 

Background of the Invention 
Modem computer graphics began in the 1960s, when a "Sketchpad" 
application program was created and its possible uses for computer aided design were 
15 demonstrated. During the past several decades, computer aided design/computer 
aided manu&cturing (CAD/CAM) technology has evolved firom only being able to 
represent two-dimensional (2D) geometry to being able to display fully shaded 
detailed three-dimensional (3D) models. With the rapid increase in computing power 
and the continuing reduction in hardware cost, CAD/CAM is being used almost in 
20 every stage of product design and manu&cturing and it has tremendously increased 
the productivity of many industries. 

However, current CAD/CAM systems are still quite limited in thdr 
capabilities. Most CAD systems are limited in the design process due to an inability to 
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enable interactive simulation and dynamic review of a product design. Moreover, it is 
increasingly apparent that there is a long felt need for a simpler way for an entire 
enterprise, i.e., management, engine^ing, manu&cturing, maintenance and suppliers, 
to view and interact with a proposed deagn for a product. Historically, when an 
5 engineer wanted to further investigate the relationships between a proposed design 
model or a procedure of how to carry out the assembly or manu&cture of the product, 
he used tools that were exten^ons of a traditional CAD/CAM system. For example, 
the engineer migiht use interactive visualization tools vMch can turn CAD data into 
^ctioning, interactive virtual products. These tools hdped the engineer to 
10 understand the fiinctionality, scale, clearances, ergonomics, and aesthetics of a new 
design. 

Several advanced 3D visualization and digital prototyping tools are available 
on the market for use with CAD/CAM technology, e.g., VisMbckUp^ from 
Engineering Animation Inc. and dVreality^ from Division Inc. These high-speed, 

IS integrated 3D visualization tools are used across the conceptual, design, analysis and 
manu&cturing phases of product development. Additionally, these visualization tools 
hdp &cilitate a concurrent engineering process and reduce the amoimt of time 
necessary to introduce new products by redudng the number of physical prototypes 
that must be created. In this way, the engineering and design teams can more easily 

20 visualize thdr products, see the effects of changes and then comnnmicate these effects 
in real time to others. Also, manu&cturers can create, interact with, share, manipulate 
and analyze new designs prior to creatmg the physical prototype. The use of 
visualization tools with CAD/CAM technology increases intmction between differrat 
groups in an enterprise and helps to reduce the total time it takes for a new product to 

25 move from an initial concept to final manu&cture. 

Although the use of visualization tools with CAD/CAM technology is 
prevalent in many industries to improve design and manufacturing methods, the 
application of VR in the field of engineering is relatively new. However, the rec^ 
development of affordable and sophisticated VR hardware, i.e. tracking devices, 

30 displaying devices and tactile devices, has fueled the o-eation of VR applications for 
improving engineering design and manu&cturing assembly tasks. 

VR is a synthetic or virtual environment that gives a us^ a sense of reality, 
even though the virtual images of the environment may or may not exist in the 
real/physical world. VR employs an immersive user interface with real-time 

35 simulation and mteractions through one or more sensorial charmels, including visual. 
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auditoTy, tactile, smell and taste. Additionally, virtual environment systems differ 
from traditional simulation syst^ns in that they are much more flexible and 
reconfigurable because they rely much less on a physical mockmp/prototype for 
creating a realistic simulation. Also, virtual environment systems differ from other 

5 previously developed computerized systems in the extent to v/bidi real time 
interaction is &dlitated, the perceived visual space is 3D rather than 2D, the user 
inter&ce may be multi-modal, and the user is immersed in a computer generated 
virtual environment. 

In the past, several attempts have been made to combine VR tedmology with 

0 traditional CAD/CAM system in different stages of produa development from 
eigonomic studies, to design, assembly simulation, tele-operation and training 
applications. However, attempts to hnprove manu&cturing planning with computer 
aided assembly planning systems have not, in general, been successfiil even when the 
design has been carried out using a CAD system. One of the main reasons for this 

5 lack of success is that assembly is depends on a great deal of e^q^ert knowledge 
which is vety difficult to formalize. Also, new products need to be more thoroughly 
analyzed for productability, quality and maintainability before committing the high 
capital required to produce physical prototypes of the new products. 

Traditional automatic ass^nbly planning methods have used the process of 

0 studying the disassembly process on the assumption that "if you can disassemble a 
part, you can assemble it, and vice versa*'. In a real-worid pineal dtuation, this may 
not be true due to irreversible fiistening processes. Also, for a given product, the 
number of feasible assembly sequences explodes exponentially as the number of 
components (parts) increases. In addition, choices of an optimal plan for disassembly 

5 may not represent the best plan for assembly. However, the present invention's use 
of VR opens up a powerfiil array of tools to solve this problem. Instead of abstract 
algorithmic assembly planning, an en^eer can perform the assembly intuitively in a 
virtual environment using VK hardware and software. Also, the infonnation 
generated by in a virtual assembly can be used for relatively predse assembly planning 

0 and verification in the real/physical world for a prototype of a new product. 

Summarv of the Invention 
In accordance with the invention, a method is provided for a virtual 
environment for sunulating the arran^g of a plurality of parts into an assembly. A 
model is created in a design environment for each part. Each model corresponds to 

S the geometry f a part and is translated into a virtual part in the virtual enviroimient 
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The design environment is integrated with the virtual environment. Each virtual part 
can be positioned in the virtual environment The positioning of each virtual part 
enables a simulation to be perfonned for arranging the plurality of parts into the 
assembly. The simulation can be modified \^ch can enable another simulation to be 
5 performed. When the modification causes a change in the virtual part, the 
corresponding model automatically includes the change to the virtual part. 

In accordance with additional aspects, the invention provides for enabling the 
automatic translation of different types of data fix)m a computer aided design (CAD) 
system to a virtual assembly design environment (VAB) system. Assembly trees, 
10 assembly constraints, and geometiy of the parts and subassemblies can be 
automatically translated firom a parametric CAD syston to the virtual environment 
provided by the Invention. 

In accordance with yet other additional aspects, the invention provides for 
enabling the creation of a realistic virtual environment with an initial location of virtual 
15 parts that can be selected by a user. Also, the user can specify the type of assembly 
environment, which can be defined in the CAD system or imported torn another 
system using any one of many standard file formats. The initial location and 
orientation of the virtual parts in the virtual environment can be spedfied by aeafing 
coordinate systems in the CAD system and transferring this coordinate information to 
20 the virtual environment. 

In accordance with still other additional aspects, the invention provides for 
creating one or more virtual hands in the virtual environment that correspond to the 
real hands of a user and which are capable of one handed and/or two handed assembly 
of virtual parts and dexterous manipulations of these parts. Li one embodim^t, one 
25 of a pair of virtual hands that are provided in the virtual emnronment can be capable 
of dexterous manipulations that are controlled with a glove virtual reality device such 
as the CYBER(3.0VE. The other one of the pair of virtual hands can be relatively 
non-dexterous and only capable of gross grabbing and manipulation movements of a 
"base" sub-assembly on to which virtual parts are to be assembled by the more 
30 dexterous virtual hand. Algorithms are used that allow the dexterous virtual hand to 
realistically grip 3D virtual parts using phyucs-based modeling and perform fine motor 
manipulations of a 3D virtual part. Additionally, the invention can produce different 
types of haptic feedbadc for a user mduding force, soimd and temperature. 

In accordance with other additional aspects, the invention provides for 
35 capturing constraint information employed by the user of the CAD system to create a 
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3D model of a part/assembly. This constraint information is employed to determine 
how the user probably intended the 3D models to be assembled. The constraint 
information is used to constrain and oreate kinematic motions for virtual parts during 
virtual assembly in the virtual enviroimient. Also, the constraint information is used to 
5 create a suggested assembly sequence of the virtual parts to the us^ of the invention. 

In accordance with yet other additional aspects, the invention provides £ot 
simulating the interaction betwe^ multiple virtual parts using constrained motions 
along determined and/or selected axes and planes. The ^ortual parts may be planar or 
axisynunetric. Also, the constraint information captured from the CAD system may 
be used to determine the axes and/or planes for constrained motion. This feature 
enables simulation of different motions such as sliding and rotating without having to 
employ computationally intensive numerical methods. 

In accordance with still other additional aspects, the invention provides for 
interactive dynamic simulation of parts in a virtual environment using physically-based 
modeling information obtained directly from a CAD system that is used to create a 3D 
model. This mformation is used to enable collision detection in real time, simulation 
of dynamic behaviors of the parts hdd in a virtual hand controlled by the user, 
dynamic interactions between the virtual hand, part(s) held by the virtual hand, a base 
assembly, objects disposed in the virtual environment, simulation of ballistic motion of 
each object in space, and simulation of dynamic behaviors of the parts while 
constrained on the base assembly. 

In accordance with other additional aspects, the inv^on provides for 
enabling a user to record the swept volume and trajectory of a virtual part as it is 
assembled in the virtual environment. The tr^ectory can be edited within the virtual 
envirorunent. Also, the swept volume of the virtual part can be viewed in the virtual 
environment. The swept vohime is created using numerical methods and this volume 
can be sent back to the CAD system. 

In accordance with yet other additioiud aspects, the invention provides for 
parametric modifications of virtual parts in the virtual environm^. Spedfic 
paramet^ for a 3D model can be tagged in the CAD system and these tagged 
paramet^ are extracted from the CAD system for display in the virtual environment 
as selectable options. AVhen these tagged parameters are selected for modification in 
the virtual environment, the modifications are sent back to the CAD system where the 
3D model of the virtual part is regenerated using all of the variational and parametric 
relations. The regenerated 3D model is r&-loaded from the CAD system into the VAE 
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system for display as a virtual part with the selected modifications in real-time without 
the user ever having to leave the virtual environment. In this way, quick design 
changes and **what-ir evaluations during the assembly evaluation process can be 
performed. 

5 In accordance with the invention, all of the above-described aspects can 

fimction individually or m any combination together. Constrained motion simulation 
is usually the de&ult mode since it is the basic fimctionality for guiding assembly 
operation. Other aspects, such as swept volume gwieration, trajectory editing, 
collision detection, design modifications, and dynamic simulation are optional and the 
10 user can switch these features on and ofif as desired. 

1 In accordance with yet still other additional aspects, the invention provides 

I forthe use of swept vohmie and collision detection together to determine whether a 
1 virtual part can be assembled safdy (no collisions) without interfering with other parts 
/ or enviroimient objects and where any interferences will occur in assembly (swept 
/15 volumes). The combined use of the swept volume and collision detection features 
/ enables a user to identify the exact instances in the trajectory path of a virtual part that 
is colliding with other parts or environment objects. These exact instances can be 
employed to identify solutions and for editing the trajectory of the virtual part. 

In accordance with other additional aspects of the invention, a system which 
20 implments substantially the same fimctionality in substantially the same maimer as the 
methods described above is provided. 

In accordance with yet other additional aspects of this invention, a computo^- 
readable medium that includes computer-executable instructions may be used to 
perform substantially the same methods as those described above is provided. 
25 Brief Description of the Drawing s 

The foregoing aspects and many of the attendant advantages of this inv^on 
will become more readOy ^predated as the same become better imderstood by 
reference to the following detailed description, when taken in conjunction with the 
accompanying drawings, wherein: 
30 FIGURE 1 illustrates a schematic overview of the usage scenario for the 

virtual assembly design environment; 

FIGURE 2 shows a schematic overview of object oriented modules of the 
virtual assembly design environment; 

FIGURE 3 illustrates a graphical user interfece in the virtual assembly design 
3 5 oivironment for a constrained motion simulation of a virtual part along defined axes; 
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FIGURE 4 shows a gr^hical user interfece in the virtual assembly design 
environment for a dynamic motion simulation of a virtual pendulum shaped part that is 
rotating and translating about a shaft; 

FIGURE 5 illustrates a graphical user interfece in a CAD environment for a 
5 swept volume with a parametric representation; 

FIGURE 6 shows a graphical user interfece in the virtual assembly design 
environment for parametric design modification options in a context menu that is 
selected by a virtual right hand; 

FIGURE 7 illustrates a graphical user interfece in the virtual assembly design 
10 environm^t for the sinuiltaneous use of sw^t volume and collision detection; 

FIGURE 8 shows an overview of two parallel axial constraints applied in a 

plane; 

FIGURE 9 illustrates a schematic overview of a scene graph for the virtual 
assembly design environment when a part is held in the palm of a virtual hand; 
15 FIGURE 10 shows a schematic overview for the alignment and mating of axis 

and plane constraints; 

FIGURE 11 illustrates a schematic overview for aligmnent and mate 
di£ferentiation of axis and plane constraints; 

FIGURE 12 shows a schematic overview for plane mating; 
20 FIGURE 13 illustrates a table that includes all pos^le combinations of axis 

and plane constraints; 

FIGURE 14 shows a schematic overview for snapping that does not destroy a 
previous constraint; 

FIGURE 15 illustrates a schematic overview for Case 1 of axis constraints on 

25 a part; 

FIGURE 16 shows a schematic ovoview for calculating the angle of rotation 
for a part; 

FIGURE 17 illustrates a sdiematic overview for Case 2 of axis constraims on 

apart; 

30 FIGURE 18 shows a schematic ovorview for calculatmg angles in Case 2 of 

axis constraints on a part; 

FIGURE 19 illustrates a schematic ov^ew for Case 3 of axis constraints on 

apart; 

FIGURE 20 shows a schematic overview for calculating translation vectors in 
35 Case 3 of axis constraints on a part; 
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HGURE 21 illustrates a flowchart for the processing and application of 
multiple constraints; 

FIGURE 22 shows an overview of the class hierarchy of constraints; 
FIGURE 23 illustrates an overview of the constraints lists included in a part 

5 object; 

FIGURE 24 shows a flowchart of the exchange of information betweai a part 
object and the constraint managei; 

FIGURE 25 illustrates an overview of a scene graph of the virtual assembly 
design system \^en a part is attached to a base part; 
10 FIGURE 26 shows an overview of the virtual assembly design system when 

the part is released in free space; 

FIGURE 27 illustrates a flowchart for the constraint manager exchanging 
information with multiple parts; 

FIGURE 28 shows an overview of swapping applied constraints and unapplied 
IS constraints; 

FIGURE 29 illustrates a flowchart for displaying constraints during the 
process of assembly in the virtual assembly design environment; 

FIGURE 30 shows the format and content of an exemplary part file; 

FIGURE 3 1 illustrates an exemplary lifting capadty data sheet; 
20 FIGURE 32 shows a graphical representation of objects sliding on a plane and 

sliding on an axis; 

FIGURE 33 illustrates a schematic overview of the allowable direction 
computation for the ctoss product of two vectors; 

FIGURE 34 shows a schematic overview for rotation of a part about the 
25 part's center of mass; 

FIGURE 35 illustrates a schematic overview for computiiig a rotational vector 
about the center of mass of a part; 

FIGURE 36 shows a graphical represoitation of a part movirig in any direction 
on a base part; 

30 FIGURE 37 illustrates a state transition diagram for a part; 

FIGURE 38 shows a graphical representation of a human eye following the 
motion of a dropping object; 

FIGURE 39 illustrates a graphical representation of an object penetrating a 
table top; 
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FIGURE 40 shows a graphical representation of an object penetrating the 
geometry of a base part resting on a table top; 

FIGURE 41 illustrates a flow chart for swept volume generation using implidt 
modeling; 

5 FIGURE 42 shows a flow chart for swept volume generation within a CAD 

system usmg implicit modeling; 

FIGURE 43 illustrates a flow chart for automatic assembly and swept volume 
generation using a UDF method; 

FIGURE 44 shows a flow chart for swept volume instance removal; 
10 FIGURE 45 illustrates a flow chart for swept volimie instance modification; 

FIGURE 46 shows a flow chart for design changes to a part within the CAD 

system; 

FIGURE 47 illustrates a flow chart for a non-parallel method of design 
modification in a virtual environment through the CAD system; 
1 5 FIGURE 48 shows a flow chart for a parallel method of design modification in 

the virtual envirormi^ through the CAD system; 

FIGURE 49 illustrates a flow chart for a parallel method of dedgn 
modification in the virtual oivironment through the CAD system using shared 
memory; 

20 FIGURE 50 shows a pseudo code fi-agment for cheddng, setting and 

processing procedures in the virtual assembly design environmoit; 

FIGURE 51 illustrates a ps«ido code fi^gment for checking, setting and 
procesang procedures ui the CAD system; 

FIGURE 52 shows a flow chart for the twirling process in the virtual hand 

25 model; 

FIGURE 53 illustrates a scene graph for the virtual assembly design 
environment when the fingers of a virtual hand are grasping a part; 

FIGURE 54 shows a graphical represwitation of finger motions for twirling a 
part; and 

30 FIGURE 55 illustrates an exemplary client computer system 

Detailed Description of th e Preferred Embodiment 
The invention is directed to a method and systen for a Virtual Assembly 
Dedgn Environment (VAE) that enables users to evaluate, analyze, and plan the 
assembly/disassembly of parts for mechanical systems. The invention employs an 
35 immersive virtual reality (VR) environment that is tightly coupled t a computer aided 
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design (CAD) system. The Invention includes: (1) data integration (two-Avay) with a 
parametric CAD system; (2) realistic 3D interaction of an avatar such as a virtual hand 
with virtual parts in the VR environment; (3) creation of valued design infonnation in 
the VR environment; (4) reverse data transfa* of the created design information from 
the VR environment to the CAD system; (5) significant interactivity in the VR 
environment between the virtual hand and virtual parts; {<$) collision detection 
betwera virtual parts; and (7) phyacal worid-based modeling of the interactivity 
between the virtual hand and the virtual parts. 

The mechanical syston of parts for an assembly is designed using a parametric 
3D CAD system such as Pro/Engineer™ In one embodiment, a user selects an option 
in the CAD system that calls the VAE system to automatically export the data 
necessary to recreate 3D virtual parts in a virtual environment Next, the user engages 
one or more VR peripheral devices to enter the virtual environment where the uso* is 
presented with a virtual assembly scene. The invention is capable of supporting a 
variety of virtual reality peripheral devices, e.g., a CYBERGLOVE by Virtual 
Technologies Inc. and a head mounted display. The various 3D virtual parts are 
initially located where they would be in a real assembly plant as defined by the user, 
which can then perform the assembly of the parts in the virtual environment. 

In the virtual environment, the user can make decisions, design changes and 
perform a host of other engineering tasks. Duriqg this process, the virtual 
environment maintains a link with the CAD system and uses the capabilities of the 
CAD system wherever requved as described in greater detail below. However, the 
operation of the virtual environment by the invention is not limited by the level of the 
intCTactivity with the CAD system. At the end of the VAE session, the user will have 
generated valued design information which is then automatically made available to the 
user in the CAD system. 

FIGURE 1 shows an overview 100 of the interactions between a VAE system 
102 and a parametric CAD system 104. In real time, tiie CAD system 104 provides 
part assembly geometiy, tolerances and part attributes, e.g., coiter of mass and 
friction, to the VAE system 102, which outputs trajectory and sequrace information 
collected in the virtual environment to another &dlity 106 for analysis. The 
ou^utted trajectory and sequence information is anployed to analyze the design £ot 
assembling the parts to determine if changes in the design assembly should be made. 
The other fiuality 106 forwards the trajectoiy and sequence information to the CAD 
system 104 along with any suggested design changes for assanbly. Optionally, the 



wo 00/38117 



-11- 



PCT/US99/30753 



VAE system 102 can provide an interfece to one or more other systems, including a 
VR based training system 108, a computer aided process planning system 1 10, a robot 
path planning system 1 12 and a specialized assembly equipment design 1 14. 

In FIGURE 2, an ov^ew 1 16 is shown of the architecture for organizing 
eight separate object oriented software modules in the VAE system 102. An 
Interaction Manager module 118 is employed to hannonize all of the modules and 
features of the VAE systm 102; and a Model Manager module 120 is used to obtam 
assembly model and environment model information from the CAD system 104. Also, 
an Output Manager module 122 is employed to create and update a graphics display 
and manage a scene graph; and a Collision Manager module 124 is used to provide 
real-tune collision detection. Additionally, an Input Manager module 126 is employed 
to obtain user input including tracking data, glove data and keyboard entries; and a 
Swept Manager module 128 is used to mate and control the editing of swept 
volumes and part trajectories. A Design Manager module 130 is employed to liable a 
user to perform parametric design modifications in the virtual environment and 
integrate these design modifications with the CAD system 104; and a Dynamic 
Handler module 132 is used to simulate dynamic behavior for a part in the virtual 
environment. 

FIGURE 3 illustrates a virtual scene 134 produced by the invention showing a 
constrained sliding motion for insertion of a virtual part 140 into a base assembly 138 
along three determined axes 136. A user controlling a virtual hand 142 can select 
colli^on options in a virtual context menu ISO and swept volume generation options 
in another virtual context menu 152. A virtual workbench 154 is disposed in the 
virtual scene 134. 

FIGURE 4 illustrates a virtual scene 144 with a pendulum shaped virtual part 
146 rotating about a fixed shaft virtual part 148 and translating along the axis of the 
virtual shaft. Disposed in the virtual scene 144 is the virtual context menu for 
selecting colliaon options in a virtual context menu 150 and the other virtual context 
menu 152 for sdecting swept volume generation options. Also, the virtual 
workbench 154 is disposed in the virtual scene 144. 

FIGURE 5 illustrates a displayed screen 156 in a CAD system for a swept 
volume 158 that has a parametric representation and which was sent back to the CAD 
system for genmtion as a feature of the assembly. 

FIGURE 6 iUustrates a virtual scene 160 of the virtual hand 142 selecting 
parameter ptions in a virtual context menu 162 for a shaft virtual part 166 that is 
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positioned in a base virtual part 164 and which is disposed on the virtual workbench 
154. 

FIGURE 7 illustrates a virtual scene 168 of a virtual part being moved along a 
trajectory path in the virtual aivironment when the swept volume and collision 
detection features are turned on. A beginning swept volume 170A and an end swept 
volume 170B for a virtual part are portioned at either end of the tnyectoiy of the 
virtual part. In the tr^ectory path, exact instances of swept volume collisions 172 
with the virtual workbendi 154 are highlighted. 

The invention can perform and/or assist a user in assembly design evaluation, 
analysis, and assembly sequence planning at all product realization stages: assembly 
plan verification (pre-product evaluation), maintwiancc verification, and alternative 
plan searching (post-production evaluation). 

In the assembly plan verification stage, the Invention enables assembly to be 
performed in a pre-defined sequence. The user can assemble virtual parts one by one 
in the virtual environment using constrained motion, swept volume and collision 
detection. If there is any interference detected during the assembly process, the user 
can try to find a way to get around it in the virtual environment. 

The m ai n tenance verification stage enables the user to check disassembly of a 
particular part. If a part needs to be taken out of a larger assembly for maintsiance, 
e.g. change a spark plug or an oil filter, the invention can be employed to ensure a 
clear trajectory path for disassembly. In one embodiment, the user removes a virtual 
part fiom its final position in a larger assembly of virtual parts and the invention 
checks for coUiaon detection during the disassembly process. In another 
embodiment, a swept volume of the tr^ectory path is created during the disassembly 
process for a particular virtual part This swept vohime is checked for interference 
with other virtual parts ui the larger assembly of virtual parts. By observiiig the 
disposition of the swept volume, the invention can detennine how much space is 
available to perform a disassembly operation. 

Sometimes it is necessary to find alternative plans or sequmces for operations 
that are already being carried out in a worktop with real parts. It is also common to 
post-evahiate an assembly op^atioiL Stopping the assembly line to perform the 
testing is not elways economically feasible and often very few ahematives can be tried 
out in the limited time available. The invention provides a viable alternative ^ere 
assembly experts can try various alternatives, choose the best one, suggest design 
changes, suggest fixturing changes and perform ergonomic evaluations f the 
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assembly/disassCTibly process. The results of these evaluations in the VAE system 
can be automatically transferred back to the original CAD system so that a user can 
quickly perform design changes without ar^ other data translation. 

From the invention test data, sevoal observations have been made: (1) Pure 
5 assembly time in a virtual environment is lower than actual assembly time (about 10- 
15%), which can be attributed to the lack of festening operations m the virtual 
^ivironment; (2) Pure assembly time for each virtual part in the assembly increases 
with the physical size of the part because the difficulty to handle a part appears to 
increase with its size; (3) Average gripping time for each part in the assembly remains 
10 ahnost the same for different sizes of parts and mainly depends on a user's practice 
and experience in the virtual envirormicnt, whmas gripping difficulty depends on the 
shape of the part (a thin, long shaft is more difficult to grab than a cubic block); and 
(4) When considering the relationship of pure assembly time and total assembly time, 
the correlation ooeffident is low for large assembly (0.9 for small assembly, 0.98 for 
15 half aze large asswnbly and 0.7 for large assembly), which indicates that human 
considerations start influencing the assembly time for larger models, e.g., movuig 
some distance to grab the part, finding a better viewing position to look at the part 
and aligning the parts. 

Besides quantitative information, the invration enables qualitative information 
20 to be obtained. For example, a fiill-size assembly in a virtual environment provides 
intuitive and valuable information that is impossible to obtain fi*om conventional 
assembly modeling by a CAD syst«n. The invention test data also illustrated other 
potential capabilities such as training, work space study and operation time study. 

With the assistance of all the capabilities of the invention, a user can perform 
25 assembly design evaluation, maintenance verification, alternative assembly plan 
searching, and part design modification as described above. Also, since the nwention 
involves the exp^ence and actions of the user, the assembly pkms generated by the 
invention automatically include input fi'om the knowledge of experienced users. 

Additionally, since the invoition is typically presorted in a fiiU immffsion 
30 mode using a head mounted display, it can be tiring to put the user in the environment 
for a long period of time. However, combining parts into sub-assemblies has been 
found to reduce the amount of time a user spends in the virtual environment. 

Virtual assembly evaluation and planning is particulariy suited for complex 
assembly operations that involve a po^a Also, automatic assonbly planning 
35 systems are well suited for assembly models with a large number f parts that require 
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relatively simple assembly operations ( involve translation and one axis rotation) 
which are often performed by robots. In some cases, a combination of virtual and 
automatic assembty evaluation can be the best solution. For example, the automatic 
assembly planning system could be used to find some feasible assembly process plans. 
5 Next, the user could then enter the virtual assembly environment (VAE) for 
evaluation, verification, consideration of practical problems related to the realization 
of the assembly design and optimization. 

In the sections below, several aspects of the invention are explained in greater 
detail, induding: (1) enhanced constrained motion simulation; (2) physically based 
10 modeling of a virtual environmrat; (3) generation of swept volumes and interactive 
swept volume trajectory editing; (4) parametric design modification; and (5) finger 
twirling. 

Enhanced Constrai ned Motion Simnlatinn 

From the CAD system, constraints are obtained and transferred to the VAE 
15 system. For axis constraints, two points in space defining the ends of the graphical line 
representing the axis are obtained. For plane constraints, three unit vectors and the 
origin defining the plane are obtained. One of the unit vectors is the normal vector for 
that plane, starting at the origin of the plane. In both cases, the ^e of constraint 
(align or mate) and the offset, if any, between the two axis or planes under 
20 consideration are also obtained. 

Additionally, during the assembly process, the geometry representation of the 
constraints of the base part and the constraints of the part being assembled are 
transformed into the same coordinate system to check for closeness. If a constraim 
meets certain oiteria, it is applied and the part's motion is limited to tiie constrained 
25 space. For example, if the part is constrained on an axis of the base part, the part can 
only slide along the axis and rotate about the axis. Alternatively, if the part is 
constrained on a plane of the base part, the part's motion is limited to the plane. 

Although the above metiiod can correctiy map the phyacal constraints in the 
virtual environment, it only works for one constraint at a time. In some cases, using 
30 the above methods will result in the loss of previously applied constraints. For 
example, in FIGURE 8, where two parallel axial constraints are to be applied and one 
has ah-eady been applied (axis Al). Simply snapping tiie constraint of axis A2 by 
translating the part will result in a loss of the previous constraint. Alb and A2b are 
two axes on the base part and Alp and A2p are two axes on the part. Alp and Alb 
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have been snapped together. A simple snapping of A2b onto A2p will move Alb away 
from Alp, which destroys the previously applied constraint. 

FIGURE 9 illustrates a scene graph 176 used to represent the graphical data 
structure of the VAE syston. The scene graph 176 provides an intuitive way to 
5 represent the hierarchical relationships between the objects in the virtual world usually 
the rdationships between differait dynamic coordinate systens (DCS). More 
importantly, it provides a way to edit and modify the relationships between the objects 
in the virtual world. For example, a part DCS 180 represents the coordinate system 
attached to a part, etc. Abase part DCS 178 moves with a FOB bird in the user's left 
10 hand so base part DCS 178 is directly under a global DCS 186. The grabbed part can 
be manipulated in the user's right hand using a CYBERGLOVE. The part's location 
and orientation is calculated through its relationship with the pahn and a FOB bird 
attached on the uscs's right-hand wrist, so that a part DCS 180 is attached to a pahn 
DCS 182, then to a hand DCS 184 before it goes to the global DCS 186. 
15 Referring to the scene graph in FIGURE 9, the following equation is used to 

transform the geometry from the part DCS 180 to the global DCS 186. The 
\partLocationXform\ is the transformation from the part DCS 180 to the global DCS 
186, the [partjnatrix] is the transformation matrix from part DCS 180 to pahn DCS 
182, the Ipalm matrix] is the transformation from pahn DCS 182 to hand DCS 184, 
20 and the [hand_matrix] is the transformation from the hand DCS 184 to the global 
DCS 186. The [baseLocationXform] transforms geometry from base part to the 
global coordinate system. 

\pco^tLocafioTiXfonn]=={part^ (la) 
(geometry of part in global) (geometry of part) x\partLocation^arm] (lb) 
25 (geometry ofbase in globaO = (geometry of base)x[Ja$eZocatfa^^ (Ic) 

Then, in the global coordinate systan, the geometry pairs of axis or plane 
constraints are compared to check for align or mate status. If certain tolerance values 
are satisfied, they are said to be constrained and equation (2) is used to apply the axis 
constraint. 

30 \pcirtjnairvc]-=[svJIe8Xform]x[cDcisRome (2) 

Where [sv NegXform] takes the part's axis to the origin of the part. Next 
[axisRotaie\ makes sure that the two axis under consideration are parallel. Finally 
\partTranslati<mXform] snaps the part's axis on to the base part's axis so that they are 
aligned and constrained. After the axial constraint is applied, the allowable motions 

35 are along that axis and about the axis. 
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\partjnatrix]^ \pjmgiTtNegXform]x[normaIRotate\ 

x{pjyrigirOffdrm]x[distancejbpjiormc^ (3) 
Equation (3) is used to apply the plane constraint where [p_originNegXfarm\ 
moves the origin of the plane on the part to the origin of the part coordinate system, 
5 [normalRotcae] makes sure the two planes are parallel. Thai takes 
the origm of the part plane back to its original portion. Finally 
[distance bp jwrmaV^orm] snaps and constrains the two planes together by moving 
the part plane in the required direction. 

In axis and plane constraints, there is axis align inserting), plane or sur&ce 
10 align, and plane or sui&ce mate, as shown in FIGURE 10. In section (a) of FIGURE 
10, axis Al (with end points Ala and Alb) is going to be aligned to A2 (with end 
points A2a and A2b). Also, in section (b) of FIGURE 10, Pl(with normal nl) is 
aligned with P2(wtth normal n2) and is mated with P3(with normal n3). 

When checking the align or mate status, differentiating them can be complex 
15 espedally for plane constraints. However, the plane normals on the part and the base 
part are in the same direction if they are aligned. An inaccurate way to differentiate is 
to check that the dot product of nl and n2 is near +1 if they are required to be aligned 
and -1 if they are required to be mated. A more accurate method for getting the 
constraint information from the CAD system is shown in FIGURE 11, whidi 
20 illustrates axis (with two end points, Aa, Ab) and plane (with an point Ori, three 
vectors el,e2, and e3) constraints. 

From the CAD syst^ for a plane constraint, a point. On, is obtained as the 
origin of the plane, as well as three unit vectors that are mutually perpendicular to 
each other. The mformation is in the part coordinate system. The corresponding 
25 information on the base part is obtained by the final transformation (the 
transformation matrix when the part is finally assembled onto the base part) between 
them. This is shown in FIGURE 12 where the plane on part (Pp) is mated with plane 
on the base part (Pb). In FIGURE 12, Pp is a plane on the part (with normal np), Pb is 
a plane on the base part(with normal nb). And nb is calculated by £quation-4. 

30 

nb = np X [TransformMaf\ (4) 

In the equation (4), [D'ansformMa^] is the transformation matrix between the 
part and the base part \^en the part is assembled to it's final location. The normal on 
35 the base part is defined in base part DCS, while the normal on the part is defined in 
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part DCS, However, when the normal vectors need to be checked, they need t be 
checked in the same coordinate system. When checking the constraints in the part 
DCS, transform nb is transformed from base part DCS to part DCS using equation-5 
(nb' is the representation of nb in part DCS). In the equation, [baselnPariXforml is 
5 the transfonnation from base part DCS to part DCS. If the part is in its final location, 
[bcMseJhPcniXfonn] is equal to [DxmsfarmMaf[^ . So there is nbp«np. 

nb in part DCS nb x [basdnPartXform] « (5) 
The normal vectors look opposite to each other, however, that is because they 
are viewed in different coordinate systems. For example, if a point is transformed to 

10 get a point in another coordinate system, when it is transformed bade, it is still the 
same point, tho^re, if viewed in the same coordinate system, e.g. in the part 
coordinate system, the two normal vectors are exactly the same. 

Therefore, when align status of two axes or two planes is checked, the dot 
product of the two axis vectors or two normal vectors are checked to be near +1. 

15 When the plane mate status is chedced, the dot product of the two normals is also 
checked to be near +1. No -1 should be involved at all 

Another useful observation can be made from the above discusdorL* ^nce the 
constraints on the base part are defined by the constraints on the part, the constraints 
on the part can be defined in an arbitrary way without affecting the final location of 

20 the part when it is assembled on to the base part. Therefore, some complicated or 
abstract types of constraints can be replaced with simple types of constraints. For 
example, a coordinate system constraint can be replaced with three axis constraints. 
This step simplifies the simulation task in some cases. 

Axis and plane (or sur&ce) constraints are the most frequently used 

25 constraints in assembly operations to fix a part on a base part or a subassembfy. In 
CAD systems, the user is allowed to pick any number of axis or plane constraints as 
long as they are not in conflict with each other. This, however, gives rise to some 
redundant information in the assembly operation. In CAD systems, the final position 
of the part is important, the order is not. However, in real and virtual assembly, the 

30 ordering of parts does matter. By analyzing all of the possible combinations of axis 
and plane constraints, invention can determine in^ch set of axis and plane constraints 
are enough and which are redundant. 

An exemplary result is listed in a table in FIGURE 13. "A" is denoted as axis 
constraint and as plane constraint. Also, numbers are employed to represent the 

35 order f the constraint. For example, "Al" means the first one applied is an axis, "P2" 
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means the second one is a plane constraint, etc. In FIGURE 13, all of the possible 
combinations that can completely constrain the motion of a part on the base part are 
listed. In the table, a symbol of "1" represents perpendicular, "//" represents parallel, 
"nJ." represents not perpendicular, and "n//" means not parallel The first column 
5 shows the various possible ways in vMch up to 3 constraints (axis or plane) can be 
used in a specific sequence to fiilly constrain the motion of a part. The second column 
shows the conditions under which a specific sequaice of constraints can fiilly 
constrain a part. 

Careful observation of FIGURE 13 leads to the three following conchisions. 

10 First, three non-redundant axis or planar constraints are sufiSdent to fiilly constrain a 
part. The user should not choose more than three axis or plane constraints in the 
assembly design. This conclusion is very usefiil and means that when the third 
constramt is applied, the part will be completely constrained. So, in constrained 
motion mapping, the invention maintains the first applied constraint wh^ the second 

15 one is q>plied. The task of maintaining the previous constraints is greatly simplified: 
the invention applies the first one using the snapping method, then uses the snapping 
method or the methods described in the next section to apply the second one, and 
when the third one is to be applied, it has readied the final location and the part is 
placed. 

20 Second, if two paralld axes are used, any third axis paralld to one of them is 

redundant. Further more, any plane paralld to them is dso redundant. 

Third, if a plane is used, any other plane paralld to it is redundant. If a plane 
and an axis parallel to it are applied, any axis parallel to the plane or the previous axis 
is redundant, and ai^ plane parallel to the plane is redundant. If two planes are used, 

25 any axis paralld to the intersection line of the two planes is redundant. 

From the three condusions discussed above, it is understood that at most three 
constraints are needed to fix the part and only the first one needs to be maintained. In 
some situations, just using equations (2) and (3) will do the work. For example, 
FIGURE 14 illustrates when PI is applied ( a plane on part Pp is snapped with a plane 

30 on base part, Pb), the snapping of the part onto A2 (snap an axis on part Ap onto an 
axis on the base part Ab) will not violate the previously applied planar constraint. 
From the analysis of all the situations in FIGURE 13, there are at least three cases that 
will need special treatment. 

Case 1: An axis constraint has been applied (Apl and Abl), another axis 

35 constraint (Ap2 and Ab2), which is parallel to the first one, is going to be applied. 
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FIGURE 15 illustrates Case 1 with an axis on part (Apl) and an axis on base (Abl) 
that have been snapped together. Another axis on part (Ap2) needs to be snapped to 
its corresponding axis (Ab2) and a simple snapping method will move Ab out from 
Ap. 

5 In this case, the snapping method is used to sn^ Apl to Abl by equation (2). 

Equation (1) is still used to calculate and check the align status for Ap2 and Ab2. If 
the condition is satisfied, the invention calculates the angle 0. 

As shown in FIGURE 16, elbl and e2bl are the end point of axis 1 on the 
base part; elb2 and e2b2 are the end points of axis 2 on the base part; and elp2 and 
10 e2p2 are the end points of axis 2 on the part. Let vector rl = e2bl - elbl, vector 
r2 = elp2 - elbl and vector r3 = elb2 - elbl. First rl r2, r3 are nonnalized; then 
angle 6 is calculated as 

e =cos"\((rlx(r2xrl)) •(rl x (r3 x rl))) (6) 
where "x" denotes cross product and "•" denotes dot product. 
15 The transformation matrix is calculated for rotating the part about axis Ab by 

angle 6, [roUite_mcarix_Ab_axis\, The final transformation of the part in the 
PahnDCS wiU be 

]part_matrix\ = \partjnatrix_A\\ x [rotate jnatrix_Ab_axis\ (7) 
where \part_matnx_Pil\ is the part_matrix calculated by using equation (2) 
20 when the first axis is applied. Also notice that Abl does not necessarily pass through 
the origin of the part DCS. 

Case 2: An axis constraint has been applied, the second one is a plane, which is 
parallel to the applied axis. As shown in FIGURE 17, an axis on part (Ap) and an axis 
on base part (Ab) have been snapped together. A plane on part (Pp) needs to be 
25 sn^ed to a plane on base part (Pb) and a simple snapping method will move Ap 
away from Ab. 

In the second case, the invention also uses the snapping method to snap Ap to 
Ab by equation (2). Equation (1) is still used to check the align status Pp and Pb. If 
the condition is satisfied, a transform matrix is formed by rotating about Ab by an 

30 angle. The angle is calculated as shown in FIGURE 18. In this figure;, elbl and e2bl 
are the end points of the first applied axis, OriPp and OriPb are the origins of the 
planes on the part and the base part, Let vector rl - e2bl - elbl, vector r2 = 
OriPp - elbl and vector r3 = OriPb - elbl, First rl r2, r3 are nonnalized. The angle 
of rotation is calculated as: 

35 e = C08"*(((rlx(r2xrl)) •(rlx(r3xrl))) (8) 
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The final transform of the part in the PahnDCS will be 
\partjnatrix] - \partjnatrixj>d'\ x [rotatejnatrix Ab^plane] (9) 
where \part_piatrix_Al'\ is the partjnatrix calculated by using equation (2) 
when the first axis is applied. Similarly notice that Ab does not necessarily pass 
S through the origin of the part DCS. 

Case 3: A plane constraint has been applied, the next one is a plane which is 
not perpendicular to the first one. If the second one is perpendicular to the first one; a 
simple snapping method can be used! 

FIGURE 19 illustrates Case 3 where a plane on part (Ppl) and a plane on base 
10 (Pbl) have been snapped together. Another plane on part (Pp2) needs to be snapped 
onto another plane on base (Pb2). A simple sm^)ping will move Ppl out of Pbl. 

In Case 3, the snapping method is used to snap Ppl to Pbl by equation (3). 
Equation (1) is used to check the align status. If the condition is satisfied, 
[p^orginNegXfbim] and [p originXform] are calculated so that Pp2 can be oriented 
15 parallel to Pb2. Now the task is to figure out a transformation vector that is parallel 
to Pbl and perpendicular to the intwsection line of Pbl and Pb2, 

The vector translation of Case 3 is shown in FIGURE 20, where the invention 
calculates the mtersection vector of the two planes by the cross produa of the 
normals of the two planes on the base part, i.e. 
20 tb»nb2xDbl (10) 

next, the invention calculates the translation direction vector t2 

tZ^nblxtb (U) 
t2 is normalized to obtain the translation vector 

tr = ((Ob2.0p2)«t2)t2 (12) 
25 The vector tr is the translation vector used to form a translation matrix 

[transkttion alongjfkme], putting the above calculations together 
\partjruWix]=={partjnatrtc_Pl]x\pjm 

xlpj>riginJffom]x[iranslation_cdmg_plcme] (13) 
where \pcp^_matnx_Pl] is the part_matrix calculated by using equation (3) 
30 when the first plane constraint is applied. 

The three special cases described above are (utuations whore spedal methods 
are needed for accuracy. In other situations, the simple snapping method can be used. 

With the help of above conclusions and methods, the mvention can simulate 
the constraints during the assembly process. The redundam constraints are processed 
35 during the constraim diecking process. A work flow chart 188 is shown in FIGURE 
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21 for processing and application of multiple constraints. In this figure, special cases 
and special methods refer to the cases and methods discussed above. 

In one embodiment, global position and orientation traddng is done by the 
Ascension Flock ofBirds™ system with an Extended Range Transmitter (ERT). This 
transmitter onploys a pulsed, DC magnetic field and is capable of detamining 6 DOF 
information fi-om each of its receivers. Three receivers are used in this system, one to 
track the head so that the user can 'look around', another to track the right hand and 
the last one is held in the left hand &cilitating assembly operations. 

In one embodiment, the CYBERGLOVE is used to monitor the finger and 
wrist movements of a user. This 22 sensor glove augments the graphical 
representation of the right hand in the VAE system. It measures the motions of the 
wrist joint, the bending of the three joints on all four fingers and the thumb, the 
abduction between all four fingers, the arch of the pabn, and the abduction of the 
thumb fixjm the pahn. The digitized output values fi-om the glove sensors are 
converted to appropriate joint angles fijr a spedfic user's hand using a calibration 
routine. These joint angles are compared against a glove tolerance to &dlitate 
releasing the part when the user stretches his/her hand to drop the part. 

In one anbodiment, the graphical basis for the invention is created with a 
Silicon Graphics IRIS Perfijrmer™ Library, IRIS Performer™ is a software toolkit for 
the development of real-time 3D graphics, visualization, and simulation applications. 
Performer™ sits "on top" of Silicon Graphics OpenGL™ libraries. It also has better 
optimization of its own functions and in turn allowed better performance when using 
complex models. 

Pro/ENGINEER™ can be used for the creation of the CAD models for use in 
the mvention. Also, Pro/DEVELOP™ is a developer's toolkit for Pro/ENGINEER™, 
which is designed to be used as a means to access the Pro/^^GINEER™ database. 
The Pro/DEVELOP™ module automates and simplifies data exchange between the 
CAD system and the VAE system. 
Constraint Management 

Object-oriented methods are used to abstract and represent the constraints in 
the invention. Humans learn about objects by studymg their attributes and observing 
their behaviors. Object-oriented programming models real-worid objects with 
software counterparts. Using object-oriented technologies, the invention can take 
advantage of object relationships wh^e objects of a certain class have the same 
characteristics Le. inheritance. Considering the constraints used in the vhtual 



wo 00/38117 



PCT/US99/30753 



-22- 



assonbly processes, even though the representations of the constraints are different, 
they all share the same behaviors: a checking process and an application process. This 
becomes a typical inheritance situation. According to the analysis and abstraction of 
real world constraints. The invention employs a Constraint class 190 and by 
5 inheritance includes other spedfic constraint classes, e.g. an AxisConstramt 191, a 
CSConstraint 193 and a PkmeConstrmnt 192 which FIGURE 22 shows in an 
overview 194. Thus, a part maintains a list of **con5traints" without needing to know 
the detailed nature of the constraint. In the Constraint dass two virtual fimctions are 
defined, checkConstraint and appfyConstraint. In the children classes, there are 

10 defined the geometrical representations according to the type of the constraint and 
override checkConstraint and c^fyConstraint according to algorithms presented in 
the previous chapt^. 

Li FIGURE 21, the assembly process is shown to be a constraint application 
process. The degrees of fi^edom of a part relative to the base part are gradually 

15 reduced as the constraints are applied. So the constraints of a part have two different 
states: already applied or going to be applied. However, some constraints are 
redundant and will never be used at all. To catch this pineal meaning, for a part 195, 
the invention en^loys three linked lists of constraints named ApphedUst 196, 
UmqfpliedList 197 and RedundantUst 198 as shown in FIGURE 23. If the part 195 is 

20 not constrained and can move in the global space firedy, all the constraints will be kept 
in the Unc^liedUst 197. If a constraint is applied, it will be moved fix)m the 
Umq^liedList 197 to iheAppliedList 196. During the assembly process, if any one of 
the constraints is found to be redundant, it will be moved torn the UnappliedList 197 
to the RedundantUst 198. After the part is fiilly constrained and placed on the base 

25 part, the UmppliedList 197 should be empty fif not empty, the remainmg dements 
must be redundant and will be moved to the RedundantUst 198). So finally in 
AppliedUst 196, there is the sequence of constraint ^plication and m the 
RedumhntList 198, there is the redundant information &om the design modd. The 
information in these linked lists provides the information on the status of the part: 

30 floating, partially constrained, or placed. In addition, the lists provide information on 
the assembly sequence. 

A ConstraintManager dass 230 can manage the constraints fox different parts. 
In the ConstrainManager 230 the invention defines three linked lists of the 
Constraint objects to hold the constraim status and data of the part that is bdng 
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manipulated. The lists in the ConstraintManager 230 provide temporary processing 
and swapping space for constraint checking and applicadoa 

The constraint mformation exchanging between the ConstrcdntManag^ 230 
and one part 195 is shown in FIGURE 24. When the user grabs a part, the constraints 
5 in the AppliedUst 196, the UmqjpIiedList 197 and the RedurtdantUst 198 of the part 
195 are handed over to their corresponding lists in the ConstrcdnManager 230 for 
handling. If the part 150 is constrained by one or more constraints, the constraints will 
be moved from the UruqjpliedUst 197 to the AppliedUst 196. If the constraint is 
redundant, it will be moved from the UruqjpliedList 197 to the RedundantUst 198. 
10 When the part 195 is released, the ConstrmnManager 230 returns the lists back to 
their corresponding lists in the part 195. Also, when the part 195 is placed, the 
ConstraintManager will give a NULL value to the UncppliedUst 197 m the part 195. 

As discussed above, the graphical structure of the system is represented by the 
scene graph shown in FIGURE 9. The part is attached to the pahn DCS 182, whichis 
15 attached to the hand DCS 184, which is attached to the global DCS 186. The location 
of the part in the global space is represented by equation (2. 1). 

[partLocatian}ffdrm]==\partjnairvc]x\i^^ (2.1) 
In the equation, {partLocationJ^orm] is the transformation from the part DCS 
180 to the global DCS 186. [part_matrix] is the transformation matrix from the part 
20 DCS 180 to the pahn DCS 182, \palm_mairix] is the transformation from the pahn 
DCS 182 to the hand DCS 184, [handjnatrix] is the transformation from the hand 
DCS 84 to the global DCS 186. In the meantime, [baseLocation^ofrm\ represents the 
transformation from the base DCS 178 to the global DCS 186. 

Jiy at some time, the user releases the part while the part is constrained, the 
25 invention wants the part to stay on the base part and move with the base part The 
relative location of the part to the base part at the time of release can be calculated by 
equation (2.2). 

\partInBaseJ^orm]=\part_matrtc]x[palm_matrtc] 
x[hand_matrix]x[baseLocationXform]'^ (2.2) 
30 At the same time, the initial scene graph is changed by moving the part DCS 

180 to attached to the base DCS 178 as shown in FIGURE 25. In this case, the base 
DCS is under the global DCS 186 whidi provides the dynamic coordinate system for 
the virtual environment scene 191. The pahn DCS 182 is attached to the hand DCS 
184 which is under the global DCS 186. Constramt handling is performed according 
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to FIGURE 24. The constraints that have been applied are stored in the AppliedUst 
196 of the part 195. 

When the user releases the part in his/her hands, if none of the constraints have 
been ^plied, the part DCS 180 will move under the global space DCS 180 where it is 
5 released, as shown in FIGURE 26. When the user later comes to r&-grab this part, the 
system needs to know '^ere the part is: in the global space or attached to the base 
part. The handling method will be difiEb^t since there is also a computation of the 
relative location of the part to the hand. 

The problem finding where the part is attached becomes easy by noticing the 
10 difference between the two situations: if the part is constrained before it is released, 
ib& AppliedUst 196 is not empty. If the part is not constrained when it is released, the 
AppliedUst 196 is empty. So whenever a part is grabbed, a check is performed 
whether the AppliedUst 196 is empty or not. TfiheAppliedList 196 is not empty, then 
that part is attached on the base part. Equation 2.3 is used to compute the relative 
IS location of the part to the palm to find out the gripping matrix. If the AppliedList 196 
is empty, then the part is in the global space as in FIGURE 26 and equation (2.4) is 
used to find the gripping matrix. 

\pcartToP<dmXform}r\partIriBaseXfo^^ 

x[hand_matrixJ^x\pcdm_?natrixY^ (2.3) 
20 \partToPalmJiformY^pca^_GlobalXfor^^ 

x[handjruitrixY^x{palm_matrixY^ (2.4) 
After re-grabbing, the scene graph goes back to FIGURE 9. 
Mechanical system assembly designs consist of many parts. During the 
assembly process, the parts can be in various stages in the environment: maybe only 
25 one axis constraint is applied, maybe only one plane constraint is applied, or maybe 
two axis constraints are applied, or m^be the part just lies on the table. So the 
constraint status of different parts are different. Whra the user grabs a part, the 
invention knows the status of the constraint infiirmation of that part: v/idch one has 
been applied; the location of the part relative to the base part, etc . When the part is 
30 released fiom the user's hand, the invention remembers the current constraint status at 
that time. When several parts are involved, the mvention keeps track of the constraint 
linked lists in every part 195 by data exchange between the ConstraintManager 230 
and the parts 195 . 

Additionally, there are bi-directional relationships between the parts 195 and 
35 the ConslrcdntManager 230. When the part 195 is released during the assembly, the 
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ConstrmntSdanager 230 returns all of the constraint lists to that part so that the 
Consirain^anager 230 can handle other parts. In this way, the invention ensures 
that the constraints of difi^erent parts are kept separate and the status of constrained 
motion of any part is maintained. 
5 In FIGURE 27, a schematic overview for the CtmstndntKdanager 230 

handling two parts (19SA and 19SB) is shown. In this figure, all of the arrows 
pointing up refer to a "when grabbed" status and all of the arrows pointing down refer 
to a '*\^^en released" status. 

In the assembly process, the user may want to reassemble a part even after it is 
10 placed on to the base part already. The user perhaps wants to tiy out some other 
sequoices, or he/she may want to assemble the part after some other parts have been 
assembled. The invention also provides the fimctionality for disassembly of assembled 
parts. 

When the invention performs disassembly, the constraints in the part need to 

15 be rearranged. When the part is placed, the applied constraints are stored m the 
AppIiedUst 196 in the order that they are applied, the redundant constraints are in the 
RedundantUst 198 and the Uruq^liedUst 197 is empty. The invention moves/sw^s 
all of the constraints in the AppUedUst 196 to the UhcppliedList 197, as shown in 
FIGURE 28. The constraints in the RedundantUst 198, howev^, need not be moved 

20 to the UmqfpliedList 197 since these constraints are filtmd out during the assembly 
process and will are not used agaia 

When the user tries to grab the part out &om the base part, the invention finds 
out where the part is. As discussed above, the invention can use the AppliedList 196 
since the list is not empty after the part is placed. The main difiference between a 

25 constrained part and a placed part is the transformation matrix that is used. In the 
former situation, the matrix is calculated when the part is released, i.e. 
\partInBaseJ^orm]. In the later situation, the matrix is the final location matrix stored 
in the Part object(fiom the original data fiom CAD system). The transformation 
matrix of the part DCS 180 to the pahn DCS 182 is calculated by equation (2.5). 

30 \partToPaJmJfform]r[flf'olLocaticn^^ 

x[hand_matrix] '^x\paJm_mcarix] (2. 5) 
Another problem in disassanbly is that when the user grabs the part, the 
system will begin checking the constraints. Since all the constraints are dose to their 
counterpart ones in the base part n^^ien the part is in the close vicinity of its final 

35 location, the part may be constrained right after the user grabs the part. This may not 
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be what the user wants to do. To solve this problem, the invention sets a time lag for 
checking for constraints if the user wants to do disassembly. The invention begins 
checking constraints five seconds after the user disassembles the part. 

If multiple parts are involved, espedally when a new assembly model is loaded 
5 in to the emoronment, the user may not know how to assemble it. Just letting the user 
tiy the possibilities makes the system unfiiendly. 

It is desirable to have a guiding mechanism in the system that can provide 
assembly instructions to assist the assembly. The instructions should be ample, 
intuitive and easy to follow. First, the user needs to know where a part needs to go 

10 onto the base part when he/she picks up the part, then he/she needs to be given 
instructions of how to assemble the part step by step. Since the user may release the 
part during the assembly process, the system needs to remember the current 
constrained status of the part. When the user re-grabs the part, the system needs to 
provide hints on the next step operation based on the constrained status. Further, if 

15 the user wants to do disassmbly, the system needs to remember the sequence of the 
previous operation and pass the information to the user to remind him/her of the 
previous operation sequence. 

To fiilfii the requirements listed above, constraint displaying functionality b 
provided. The geometry of the constraints are displayed when the user grabs the part: 

20 for axis, a line is displayed; for planes, a rectangle near the contact is displayed. When 
several constraints are involved, different colors are used. This gives the user a very 
intuitive feel for the assembly process. Further more, the constraints are displayed 
according to the status of the constraints. If one axis constraint is applied and the user 
lets the part follow the base part, next time when the user grabs the part again, the 

25 applied axis will not be displayed. If a redundant constraint is detected, it will not be 
displayed anymore. When the part is taken away from the base part, the next time 
when the user wants to reassemble it, all the constraints come back again except the 
redundant ones. 

Although the requirements of the guiding medianism are complicated, the task 
30 is not that con^)lex because the invention recalls the information stored in the 
constraint lists. The metiiod of handling tlus task is to make use of the constraint lists, 
the AppliedUst 196, the Uncqjptiedlist 197 and the RedtmdantUst 198. Whenever 
the invention employs a displayer 232 to display the constraints, it starts with the 
XJmqrpUedUst 197. This ensures that only the unapplied constraints are displayed. The 
35 number f constraints displayed is reduced as the part is being assembled, which 
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means that the allowed degrees of freedom reduces. Also, since the invention 
processes the redundant constraints all the time, if the constraint is moved to the 
RedundantList 198, the invention will not get it later, so gradually only the valid 
constraints are displayed even if the user starts all over. An overview of the guiding 
5 mechanism discussed above is shown in FIGURE 29. 

A detailed scenario is presented above for managing constraints in the virtual 
assembly environment. The system can efiSdently manipulate multiple parts for 
assembly evaluations. When several constraints need to be applied, all of the 
constraints are applied in conjunction with the previous ones. When multiple parts are 
10 involved, each part moves observing its own constraint set. 
Physical Based Modeling 

The scene graph method provides an intuitive way to represoit the hierarchical 
relationships between the objects in the virtual world (usually the relationships 
between dififerent dynamic coordinate systems). More importantly, it provides a way 
15 to edit and modify the relationships between the objects in the virtual worid. 

One important feature of the invention is constrained motion simulatioa The 
constraint information is extracted from CAD system and each independent constraint 
satisfied will reduce the number of allowable movements of the objects rdative to 
each other. The invartion can simulate axial and planar constraints during assranbly 
20 design process in any kinds of order and combination. The invention employs methods 
that can dmulate physical constraints conmionly used in assembly design without 
u^ng computationally expensive collision detection. 

In phy^cally based modeling, the basic equations of motion for rigid bodies 
used to set up the emulation model are the Newton-Euler's equations, which are as 
25 follows: F = MV* andN = dL/dt = Ia)* + ©xL 

Where F is the external force; M is the total mass of the system; y is the Hnear 

acceleration of the center of the mass of the system; dL/dt is the time derivatwe of 
angular momentum in the space frame( which is equal to external torque N); I is the 
3x3 inertia matrix and <d' is the angular acceleration; ©xL is the cross product of 

30 angular velocity vector and angular momentum vector. In order to solve for the 
acceleration, velocity and displacement of the system, the mass properties are needed, 
Le. mass and inertia matrices of the part or the system. 

The invention gets around calculating mass properties of polyhedral objects by 
getting the information directly from the CAD system y/ken the model is dedgned. 

35 The mass and inertia matrices are defined (unless the object is broken or deformed) 
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once the model is designed. After investigating the available infonnation which can be 
queried from CAD systems (e.g. ProEngineer'^), the devdop^s toolkit (e.g. 
ProDevelop^ can be used to extract the information. When the model geometry and 
constraint information are written out, the mass properties are writtm into a property 
S file for each part (or subassembly if subassemblies are used) of the model. The file 
format and content are illustrated in FIGURE 30. Note that in the exemplaiy property 
file, the invention also includes information other than just mass properties such as 
units, sur&ce finish, tolerance values, sur&ce area, density, and volume. 

When the invention loads the model into the virtual environment, it also loads 

10 the property of the parts or subassemblies at the same time. The information can be 
quoied fit)m the part whenever it is needed during the simulatioa 

Assembly models differ tremendously in terms of size and numbers of parts, 
from tiny motors to large aircraft. In the assembly operations for the different models, 
human fimctionality is different. For some small assemblies, assemblers may use their 

IS bare hands with assistance from tools. For large assemblies, they depend on tools, e.g. 
hoists, to lift some big parts and put the parts in thdr final locations. 

In the VAE system, this fiict is taken into constderatioa It is easy to use one 
hand to grab and lift a several himdred pound truck engine in the virtual environment. 
But this will result in loss of feeling of realism, or even trust in the system. So the 

20 inention distinguishes and categorizes the assembly modeb according to human 
being's behaviors and abilities. 

The criterion that the invention uses is the strength survey data of human 
beings. For workers on the assembly lines, if he/she can lift the part with one hand or 
both hands without difficulty, he/she will lift the part and cany the parts to the 

25 assembly. This comes &om the observation of real world op^ations and from the 
concerns of productivity of industry. The invention can categorize a part into three 
categories by it's wdght: (1) being able to be lifted by one hand; (2) being able to be 
lifted by two hands; or (3) need to be lifted by a tool. If the part can be lifted by one 
hand, when the user tries to grab the part, he/she can grab it and move it normally. If 

30 the part needs to be lifted by two hands and the user tries to grab and lift it with only 
one hand, the invention can inform the user that the part is too heavy for one hand 
lifting and suggest he/she lift it with two hands or g^ help firom some tools. For parts 
that are too heavy to be lifted by assembler's bare hands, the invention can notify the 
user to use a tool. Although this kind of categorization is crude and simple, it can 

35 represent the real world situation. One interesting observati n is that novice users 
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tend to reach out his/her hands to pick up relatively small parts even before any 
explanation is provided on how to grab the parts in the environment. If he/she is put 
into the environment with a large part in fi^nt of him/her, the user usually stays static 
and waits for instructions. 
5 FIGURE 3 1 shows a listing of the lifting capacity for infrequent movonent 

and short distances data used for one onbodiment. Although the data for both men 
and women is provided, this embodiment uses the figures for women to make sure the 
system works for everyone. If the part is bdow 20 poimds» the invention uidicates 
that the part can be lifted by one hand; if the part is between 20 and 40 pounds, it 

10 indicates that the part can be lifted by two hands; beyond that, the part is put in the 
category of "needs to be lifted by tools". 

As described above, constrained motion simulation is used to dniulate physical 
constraints in the virtual environment. Although the invention can simulate physical 
constraints by constrained motion without using collision detection, collision detection 

15 is still a oitical aspect to verify and validate assmbly sequences and otdm. Furth^, 
since the invention can sunulate dynamic behaviors of the parts in the virtual 
environment, the invention can be used to determine if these behaviors improve the 
reality feeling in the virtual environment and help the assenibly planning process. 

The simple categorization of the parts in the assembly models enables the 

20 invention to define the scope of dynamic simulation of the parts in the virtual 
environment. In one embodiment, the invention implements dynamic simulation in 
cases where the models are small and the parts are not heavy, i.e., in the range of 
"being handled by one hand". For larger modeb and parts, it is not applied since these 
kinds of behaviors and motions are not allowed in real industrial world anyway 

25 because of safety concerns. 

During the assembly process planning of operation, certain behaviors such as 
object bouncing are not of major concern because the invention can assume the user 
will behave rationally in the assembly operation. He/she may hit a part with a hammer 
to adjust its shape, but will not urmecessarily hit a part with the base part or other 

30 parts. The invention can model the behavior of the part in the user's hand and on the 
base part. In the virtual envirormient, first time users may try to throw a part away to 
see what a virtual reality system is, but an experienced user who wants to verify 
his/her design would rarely behave in this way. Thus, the Invention provides models 
for dynamic behaviors on the part wMle the part is held in the user's hand and while 

35 the part is constrained on the base part. 
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There are three kinds of physical motions for an object discussed in detail 
below: (1) free motion in space; (2) translation on a plane and along an axis; and (3) 
rotation about an axis. 

Free motion in space of an object is the simplest phyacal motion to model.. An 
5 object just follows a ballistic trajectory as desoribed in elementary phymcs texts. The 
equations of motion are shown in equations 3.2.1 and 3.2.2. In the equations, t is the 
time of motion, Vo and <Do are the initial linear and angular vdodty vectors, SO and S 
are initial and instantaneous position vectors, and finally, Ango and Ang are initial and 
instantaneous angle values of the object's local coordinate system relative to the global 
10 coordinate system. 

S= SO + Vo*t + 0.5*G*t' (3.1) 
Ang = Ango + ©e * t (3.2) 
Since the Invention only needs to obtain position and orientation values to 
update the display and the values can be computed directly with equations 3.1 and 
15 3.2, the Invention does not need to do any integratioa The critical issue here is how 
to obtain the initial linear and angular vdodty vectors. 

Before the object can move fredy in the global space, the part is dther hdd in 
the user's hand or constrained on the base part. For simplidty, the Invention keeps 
track of the object's global positions and orientations with respect to time no matter 
20 where the object is. The global information of portion and orientation of the object is 
represented by a transformation matrix. Referring to the system graphical scene graph 
in FIGURE 9 discussed in detail above, equation 3.3. can be used to compute the 
transformation matrix if the part is hdd in the user's hand. In the equation, 
\partLocationJiform'\ is the transformation from part DCS to global DCS, 
25 \partjnatrix] is the transformation matrix from part DCS to pahn DCS, 
[palmjnatrix] is the transformation from pahn DCS to hand DCS, and [hand_matrix] 
is the transformation from the hand DCS to the global DCS. 

\partLoccaionXformy^\panjnatrt^^^ (3,3) 
If the part is constrained on the base part before it can do global free motion, 
30 then the scene graph in FIGURE 9 is modified to the one shown in FIGURE 25. The 
Invention uses equation 3.4 to calculate the global transformation matrix in this 
situation. In the equation, the \partjnatnx] is the transformation matrix Scorn part 
DCS to base DCS (this matrix is different from that in equation 3.3). The 
[baseLocatiorOfform] is the transformation matrix from base DCS to global DCS. 
35 \pcirtLocation)^crmy=\partjnca^^ (3.4) 
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At the moment wh^ the object is able to move freely in space, two 
neighboring instances are diosen (an object in a certain fi:ame is called an instance) 
and calculate the initial vdocity vectors based on the differences of positions and 
orientations of those two instances ( Pi, Ai are the position and ori&ttation vectors of 
5 the first instance and Ps, As are the position and orientation vectors of the second 
instance), as illustrated in equations 3.5 and 3.6. This is just an approximation method 
since the Invention can use finite translation and rotation displacement to calculate 
velocities. However, since the time difference between two neighboring fiames is 
usually very small, the approximation can still provide a good initial condition for the 
10 free motion of the object. The positions and ori^itation vahies are computed by 
solving an inverse transformation problem, i.e., compute the translation values and 
rotation angles fit)m a transformation matrix. 

Vo= (P2-Pi)/(t,.ti) (3.5) 
©0= (A2-Ai)/(t2-ti) (3.6) 
15 When the part is constrained on the base part, if the part is constrained to 

move on a plane, the part can only slide on the plane. If the part is constrained by two 
parallel axes, or one axis and a plane parallel to the axis, the part is only allowed to 
slide along the axis. If the part is constrained on two planes, the part can only move 
along the int^Bection line of the two planes. The base part is manipulated in the left 
20 hand of the user and the movement direction of the part may be changing with respect 
to the global frame. Also, FIGURE 32 illustrates an exemplary object sliding on a 
plane or sliding along an axis. 

For these situations, the Invention sets up a vector called 
"AllowableDirection" to represent the aUowable translation direction as shown in 
25 FIGURE 33. In this figure, endl and end2 are the two end points of an axis, n is the 
normal vector of a plane and G is the gravity acceleration vector 

This vector is computed for different situations as illustrated in equation-3.7.I 
and equation 3.7.2. For sods movement, "AUowableDimtion" is along the axis. For 
a plane movement, "AllowableDirection*' is a vector that is perpendicular to the 
30 plane's normal vector. 

AIlowableBirection = «id2-endl (3.7.1) 
AOowableDirection » n x ( G x n) (3.7.2) 
The symbol "x" represents cross product of two vectors. If the angle between 
G and AUowablelMrection is greater than 90^, the Invention negates 
35 AOowableDirection and normalizes AllowableDirection for later computation. 
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When the Invention computes AllowableDirection, endl, end2 and n in 
FIGURE 33 should already be transformed to the global coordinate system in order to 
compare with G. The original data are obtained in the part's local coordinate system. 
The Invention transforms them from the local DCS to global DCS u^ng equations 
5 3.8.1 and 3.8.2. ^pca^lLocatiorCCform] is the transformation matrix calculated from 
equation-S. \partLocationXform JdrVector] is the same transformation matrix except 
the translation values are set to be zeros because endl and end2 in part DCS are 
points while n in part DCS is a vector. 

endl,2 = (fflidl,2 in part DCS) * \partLoccaionXfarm] (3.8. 1) 

10 n = (n in part DCS) * \pariLocatioiiXform JorVector] (3.8.2) 

The part may not be able to move if the Invention takes static friction into 
account, even if there is a direction to move. Suppose the static friction coeffident 
between the part and the base part is fs» the condition for the part to be able to start 
moving is checked with equation 3.9. After the motion begins, dynamic friction 
IS coeffident fd (which is smaller) is used to get the acceleration a by equation-lOa. In 
the equation, p is the angle between G and AllowableDirection, m is the mass of the 
object, |G| is the magnitude of G. 

Angle (between G and AUowableDirection) < 90-Wi\&) (3.9) 

In this situation, the equations of motion are described in equations 3.10.1-4. 
20 In these equations, a, V and P represent the acceleration, velodty and position of the 
object. Notice that AUowableDirection is changing with the movement of the base 
part, the position of the part is actually obtained by simple numerical integration using 
Euler's method. 

a = F/m = (m|G|cos(P) - fd*m|G|sin(P)) / m * AnowableDirection 



25 =(|G|(cosO) - £a*sin(P)) • ADowableDirection (3.10.1) 

V^i = V, + a*t (3.10.2) 

Pi^i = Pa + Vn * t + 0.5*a*t* (3. 10.3) 

«IP = P».,-P„ (3.10.4) 



The vector dP will be used to form a transformation matrix [translate by dP] 
30 to update the position of the part. But before doing this, the Invention transforms this 
vector to the base DCS from the global DCS by equation 3.11 since all the 
computation is done in global DCS. The new part_matiix (the transformation matrix 
of part DCS in base DCS) is calculated and updated by equation 3. 12. 

(dP in base DCS) = dP ♦ {baseLocationXformJorVectorY^ (3.11) 
35 [new j>artjrtatrix]-\partjnatrix]xli^^ (3. 12) 



wo 00/38117 



-33- 



PCT/US99/3Q753 



For the second kind of physical motion, the part is constrained on the base part 
by an axis, the part will tend to rotate about the axis if the center of mass of the part is 
not on the axis. In this case, first, the invention transforms endl, end2 and CM (the 
center of mass) fi-om the part DCS to the global DCS using equation 3. 13. The vector 
S RotVec (the vector and object is rotating about) and CMVec (the vector that passes 
CM and perpendicular to RotVec) and can be obtained firom equation 3.14 and 
equation 3.15. 

(endl.2; CM)sfMi = (endl, 2; CM in part DCS) » IpcaiLocationXform] (3. 13) 
10 CMVec = (end2-endl) x (( CM - end2) x ( cnd2 - endl)) (3. 14) 

RotVec = G X CMVec (3.15) 

FIGURE 34 illustrates rotation about an axis with a local coordinate systm at 
the origin of the center of mass having an orientation that is defined i^en the part is 
15 designed in a CAD system. 

In this particular situation, Euler's equation can be simplified to equation 3. 16, 
where Jaxis is the moment of inertia of the object with respect to the axis of rotation, 
a> and a are the angular velocity and acceleration, m is the mass, |G| and |CMVec| 
are the magnitudes of vectors G and CMVec, p is the angle between vector CMVec 
20 and RotVec shown in FIGURE 34, and fi^ is the rotational fiiction coefficient For 
simplification, the fiictional torque is represented as &*a>, 

Jaxis ♦ a = Torque m * |G| * |CMVcc| * 8in(P) - fi-*® (3.16) 

Or 

a = Torque/ Jaxis = (m*|q*|CMVec|*sin(p)-fi^a)) /Jaxis (3.17) 
25 In equation 3. 17, G is constant vector, CMVec and p can be easily calculated, 

m can be queried fix)m the part directly, so the only thing left is to compute Jaxis. The 
extraction of mass properties of the part, induding the inertia matrix Icm with respect 
to center of mass is discussed above. Icm is a second order tensor To calculate Jaxis 
fi'om Icm, the first step is to create another coordinate system, with the origin still at 
30 the center of mass, and one axis ( z') parallel to RotVec as shown in FIGURE 34. The 
next step is to find a transformation matrix that relates the two coordinate systems. 
This matrix, T, can be fonned by rotating z to z*. The new inertia matrix of the 
object, Icm', with respect to the new coordinate system x'-y'-z' can be obtained by 
equation 3.19. 

35 Icm'^T^Icm*!* (3.19) 
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Let z' be the axis f rotation, the moment of inertia about z* is just I^z*, or 
Icm. Using general paralld axis theorem, with dist calculated from equation 3.20, 
Jaxis can be computed from equation 3.21. FIGURE 34 ilhistrates x-y-z as the 
original coordinate systm with Icm and i -y'-z* as the new coordinate system with z* 
5 parallel to RotVec 

dist - ((endl-end2) x ((cndl-end2)x(CM-endl))XCM-endl) (3.20) 
Jaxis = 1^7* + m* dist^ (3.21) 
With Jaxis conqnited, the Invention employs equations 3.22.1, 3.22.2 and 
3.22.3 to integrate the rotation angles of the object about RotVec. In equation 
10 3.22.1, a is the angular acceleration computed in equation 2.17, co. and An are initial 
angular velocity and an^es for each int^ration step. 

©n+l = ©□ + a * t (3.22.1) 
An+1 = A„ + ©n • t + 0.5*a ♦ t* (3.22.2) 
15 dA«A-A9 (3.22.3) 

Finally, dA b used to form a rotation matrix to adjust the old part 
transformation matrix in base part DCS. The rotation axis, RotVec does not 
necessarily pass through the origin of the part DCS, The transformation matrix 
{rotation_dAjib<mt_RoiVec\ is a matrix combining a translation of endl bade to 
20 ori^ a rotation matrix, and a translation of endl from origin to its initial position. 
The new matrix is calculated in equations 3.23.1 and 3.23.2. 
[rotation_dA ab{mi_RotVec]-[trcm_endJ_origin] x 
[ roiation dA] x [trans endJ origin back] (3 .23 . 1) 

[new j>coijttaira]=fpcfftjmitrt^^ (3.23.2) 
25 When a part moves in space, or moves on a base part, the part should stop 

moving if its motion is blocked, e.g., stopped by the table, or stopped by the base part 
geometry. As mentioned before, since a purpose of emulating dynamic behaviors is 
to assist assembly operation, the Invention does not pay much attention once the part 
is out of the us^s 'Mrtual hand" or is away from the base part. For the fonner case, 
30 the part stops movmg if the part hits the table or other environment objects and the 
Invention does not go fiirther to figure out the balanced resting position or orientation 
for the part on the table. This short cut saves computation time and lets the Invention 
concentrate on interaction issues. 

The situation is complicated if the part moves on the base part, which is 
35 illustrated in FIGURE 36. In this figure, the part can move in any direction on the 
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base part and PI and P2 are planes (or geometry) on the base part. The part is sliding 
on a plane PI on the base part. If the part moves in the direction of tl, collision 
detection is used to check \^ether the part is still touching the base part. If the part 
slides away from the base part, it goes to free space. If the part moves along t3, 
collision detection is used to check if the part will be blocked by F2. If the part 
moves along t2, it is unclear which situation will occur first so both situations are 
checked. This brings up a conflicting collision problem- a simple report whether the 
part is colliding with the base part is not enough. If the part is moving along tl, 
collision detection of the two touching sur&ces is performed; if the part is moving 
along t3, the collisions of the part other than with PI should be determined. The same 
situation will occur when a shaft is inserted into a hole, the part may be sliding out of 
the base part or blocked by other geometry other than the hole. 

In one embodiment, a RAPID^ collision detection &dlity developed in at the 
University of North Carolma at Chapel Hill is used. The fecility requires triai^gular 
data of the model as well as the position and orientation of the model in the global 
space. The frtdlity can report the collision status (coUiding or not) and the 
intersecting triangle pairs between two models accurately. It supports the option of 
reporting the first collision or report all of the collision pairs. A direct way to solve 
this problem is to let RAPID find all the colliding triangles, and distinguish the 
interfering ones on PI with those on P2. However, this is not a feasible solution for 
seva^ reasons. 

First there may be many holes on PI and this will result in a great number of 
triangles on PI . Looping through all the triangles will be time consuming. Second, due 
to floating mmiber errors, RAPID may not find all the touching triangles (this is not a 
problem of RAPID itself this is usually caused by the series of transformation matrix 
multiplication). And thirdly, asking RAPID to report all the collision pairs is much 
slower than reporting the first collision (about a &ctor of 10). 

The solution is to modify RAPID to solve this spedfic problem. Since the 
Invention mainly focuses on planar and axial constraints, RAPID was modified to 
solve the coplanar or coaxial interference checking problems. Besides its own opdons 
to perform different levels of con^tation, "FIRST_CONTACT'' and 
•'ALL_CONTACT", another set of options for coplanar and coaxial atuations were 
added: "KEEP COPLANAR" (report if the triangles are parallel and the distance 
between them is less than a tolerance value), "SKIP^COPLANAR" (do not report if 
the triangles are coplanar and very close), "SKIP^CYUNDER" (do not report if it is 
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a equal-diameter coaxial insertion), "NOCARE_COPLANAR" (normal mode, do not 
consider coplanar or coaxial problems). When the part moves on the base part, the 
modified RAPID &dlity perfonn two collision detection checks: one for cfaeddng if 
the part is still touching the base part and another one for checking if the part is 
5 blocked by geometry of the base part other than the plane or the cylinder the part is 
sliding on. Since the first check will always detect the collision if the part is touching 
the base part, the "KEEP^COPLANAR" option is selected. Also, since the second 
check vfUl always ignore the collision between the touching triangjles, 
"SKIP^COPLANAR" or "SKIP_CO AXIAL" options are employed. 

10 In FIGURE 36, if the part moves along tl. the first check will tell whether the 

part still touches the base part; if the part moves along t3, the second check wll notify 
whether the part is blocked by P2; if the part moves along t2. whidiever of the two 
check is first will put the part either in space or on the base part. 

With the collision detection problem resolved, the interaction issues in the 

15 virtual assembly environment can be analyzed. In the virtual environment, there is the 
user's virtual hand(8), the virtual part, the virtual base part, virtual tools, and virtual 
environment objects. Since how the virtual part is being handled and assembled is 
part of the Invention, the part is the center of the whole system. The xiscr can grab or 
lift the part with his/her bare virtual hands or with some virtual tools, so the user is the 

20 dedsve &ctor for the virtual part and the virtual tools. If the user uses a virtual tool 
to manipulate the virtual part, the virtual part should observe feasible motion defined 
by the virtual tool 

Different state variables are used to define the status of a virtual part in the 
virtual environment. The slates are: INHAND (grabbed or lifted by the usex), 

25 ONTOOL (manipulated by a virtual tool), ONBASESLIDE (constrained on base and 
can move relative to the base part), ONBASESTATIC (constrained on base and 
cannot move relative to the base part), INSPACE (fiw moving in space), STATIC 
(remaining in the same position and ori^tation), and PLACED (placed on the base 
part in the final assembled location). The virtual part will be handled according to 

30 different states. If the virtual part is INHAND, the motion of the part is totally 
dedded by the user's virtual hand. If the part is ONTOOL, its motion is decided by the 
virtual tool, whose motion is decided by the user. If the part is ONBASE or 
INSPACE, dynamic simulation methods discussed above will be used to detennine the 
motion of the virtual part. If the virtual part is STATIC, no action is performed. 
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The States of the virtual part can change from one to another. A transiti n 
state diagram 234 is shown in FIGIXRE 37. which is used to demonstrate the changes 
of the state of a virtual part and the possible causes of these changes. The state 
diagram 234 also shows the interactions between the user's virtual hand(s), the virtual 
5 part, the virtual base part, the virtual tools, and the virtual environment objects. Also, 
this state diagram 234 provides a conveniwit way to handle the part in diflFerent 
situations. Whenever an interaction occurs, the stote of the part is updated, the part is 
thai handled according to its previous state and the current state. 

If a part moves freely in space, it will foUow a sin:^)le ballistic trajectory as in 
10 equation 3.3. Once the initial conditions are known, the only thing that changes the 
motion of the part is its gravity with acceleration G, ^^ch is 9.8 (meter/second^ or 
385.8 Cmch/second^. Surprisingly, to let the user fed that the motion of the part is 
realistic, G is scaled down to a smaUer value using a scale &ctor 0.2-0.3. The scale 
factor can be determined by trial and error. 
15 Another ima^sting and conflicting observation is that when the integration for 

a virtual part rotates on the virtual base part, there is not a need to scale G down 
when calculating angular acceleration using equation 3.17 in order to OTble the us«r 
to feel that the rotation is realistic. 

The explanation of this phenomenon can be found in the nature of the 
20 immerave virtual environment. In real worid space, humans usuatty use their eyes to 
follow the moving objects and human's eyes can follow a moving object even when 
the object moves with an acceleration greater than gravity. Ahhough in the real worid 
human eye movement can be as fest as 600 degrees/second, in a virtual environment 
the human being's viewing update is usually determined by the movement of a tracking 
25 device attached to his/her head, instead of his/her eyes. Even if the gi^hics of the 
system can be updated 30 frames per second, the movement of the human's head, 
espedally rotation, is limited to peak vdodties of about 600 degrees/second for yaw, 
and 300 degrees/second for roU and pitch. In a My immersed virtual environment, the 
user usually wears a head-mounted-display device, i.e., a hehnet. The weight and 
30 inertia ofthe hehnet further limit the motion ofthe user's head. 

Suppose the object is 1 meter away from the user's eyes. If the object is 
dropping with normal gravitational acceleration as can be seen in FICfURE 38. the 
head of the user needs to rotate with an angular acceleration computed in equation 
3.24. Tliis requires the human's head needs to rotate 280 degrees in one second ( 0.5* 
35 otcj^ ♦ t^. This is close to the peak abiUty of head movement (300 degree/second) 
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without a helmet. It is impossible for the user to rotate his/her head in the virtual 
space with a hehnet on his/her head to fiilfil this requirement. In FGURE 38, a 
schematic overview is shown of a human eye following the motion of a dropping 
object. 

5 ct^ - |G| / d = 9.8 / 1 = 9.8 (rad/sccond^ « 561 (degrees/second^ (3.24) 

If the scale &ctor is used (about 0.250, the motion requirement becomes 
280*0.25 = 70 (degrees) in one second, which is a quite reasonable number. 
Therefore, the appropriate gravity acceleration in virtual space is determined by the 
human &ctor, i.e., the ability of human's movement in the virtual environmoit. This 

10 also can explain why the gravity acceleration needs not to be scaled down for rotating 
objects: the rotation of the object is usually a local motion, the position of the object 
in space does not change much and the user does not need to move his/her head to 
follow the motion of the object. 

When a virtual part drops down &om the user's hand or from the base part, it 

15 may be stopped by a virtual table and stay on the table, or it may also Ml down onto 
the ground in the virtual environment. It is v^ inconvenient and sometimes even 
bothersome to go to grab the part again from the floor. In this case, the Invention can 
let the virtual part go back to its original portion and orientation when the virtual part 
reaches the floor in the virtual environment. 

20 When a virtual part is stopped by a virtual table or other virtual environment 

objects, the part's state changes from INSPAC£ to STATIC. At that time, the virtual 
part may be penetrating into the virtual table, as shown in FIGURE 39. This is the 
most basic problem in traditional physically based modeling syst^ns. Usually, the 
object is moved back to its position and orientation of last frame and moved with a 

25 smaller step until the exact contact point is found. To get around this problem, the 
object is moved back to its position and orientation of last frame, i.e., when it is not 
colliding with the table. 

However, the above trick can not be used when the part is stopped by the base 
part geometry since the user can view the part from different aiigles, as shown in 

30 FIGURE 40. In this case, the object is moved back to the position and orientation of 
i the last frame and the linear and angular velocity vectors are set to zero. So the 
integration will start from zero velocities and finally it will remain in a location that is 
vecy dose to the bloddng geometry of the base part. The result of this method is that 
the part slows down when it hits the base part. 
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Since there can be multiple virtual parts in the virtual environment, every part 
may be in its own different state at the same time. To handle all of the parts correctly 
and conveniently according to its current state, a traversal fimction goes through the 
parts and takes corresponding actions according to their respective states. 
S In equation 3.10 and equation 3.22, the most ^proximate integration method 

is used. Although its accuracy is of 0(h2), practically it is good enough. The reason is 
that the absohite positions and angles are not critical and the approximation is enough 
as long as the motion looks correct For example, it is difficult to tell the difference of 
several degrees in the virtual space v^en a part is rotating. If the part follows a 

10 pendulum like motion, there is a tendency to believe the motion is correct. 

An important aspect of physically based modeling m virtual assembly is to 
assist in simulating the design intent of the designer of a part. Overall, constrained 
motion simulation is the convenient way to achieve this goal. The constrained motion 
methodology aligns the concepts of constraints in the assembly de^gn with the actual 

15 assembly operation. When physical constraints are simulated, not only the designer's 
intent is represented, but also the processes in which the assembly is physically carried 
out are shown. This aspect of the invention also provides a direct and intuitive way of 
assembly evaluation ance the VAE system is simulating the physical assembly 
process. This simulation can be used for all sizes of assembly models and is 

20 conqnitationally effective since it avoids extensive collision checking and motion 
calculations in eveiy ffame. 
Swept Volume Generfttign and Trajofftgry 

The invention provides for generating swept volumes directly in a CAD system 
(e.g., ProBngineer™). Also, real-time assembly tnyectory and swe^ vohmie editiiig 

25 is provided in the virtual environment by the invention. The swept vohmie instances 
cq)ture the position and orientation of the moving object in each frame, whidi can be 
picked, r^oved and modified until the tnyectory is satisfactoiy to the user. The 
trajectory is sent to a CAD system by the invention where it is used to generate the 
swept volume. The swq^t volumes can then be taken back from the CAD system into 

30 the virtual environment for fiirther evaluation. Swept volumes genoated in this way, 
are accurate concise, and easy to process in the CAD system. 

The transformation matrices information obtained fit)m the coordinate system 
of the base part or the global space becomes meaningless outade the virtual 
environment. The real issue is the relative information between the swept volume 

35 instances thmselves. As a conveni^ referrace for the swept volume instances, the 
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first instance is picked as the reference for all of the other instances. Suppose there 
are Tl, T2, T3, etc., as the transformation matrices. The relative transformation 
matrices of the instances to the first instance can be obtained as T2T1'*, T3T1'\ 
T4T1 ^ etc. 

The final problem is to find the translation values and rotation angles, given a 
pure rotation and translation matrix. This is a typical transformation bverse problem. 
The translation elements are relatively easy to calculate. The rotation angles are not 
unique because they depend on the order of rotations performed about the X, Y and Z 
axes and some special cases. If the angles are computed using a combination matrix 
composed by rotations in a certain order, e.g., first rotate by Y, then Rotate by X, and 
finally rotate by Z, this order is kept until later when the matrices are created. 

To generate swept surfaces and volumes using an implicit modeling approach, 
the geometric model and a path describing the part motion (called ST) need to be 
defined. The model geometry is represented by a file in stereo-lithognq}}^ format. 
The fiiie can be easily created in CAD systems where the part or object is designed. 
The path is defined by a series of transfonnation matrices (Tl» T2, ...). The 
transformation matrices are defined in the global coordinate system. The geometry of 
the part is used to generate an implicit model in the form of a volume, called, VI; 
Another volume which can strictly bound VI as it moves along the path ST, called 
Vw is also constructed. As VI is swept through Vw by moving in small steps. Ax, 
the Boolean operation is used to sample VI in Vw. Finally all of the samples of VI in 
Vw are extracted using a contouring algorithm to form the boimdaiy swept volume. 
The output of the algorithm is also triangles that approximate the swept vohmie 
surfiice. The data size is usually very big. A triangle decimation method is then used 
to reduce the number of triangles. A flow chart 236 of the implicit modeling 
algorithms is shown in FIGURE 41. 

Although the algorithm is robust to handle abnost every kind of geometry, the 
problem is that the data size of it's output is too big and the accuracy is rdativety low. 
Its' application is limited in cases where the accuracy requirement is low. The error is 
first introduced when the part geometry is triangulated. Then more errors occur in the 
working cell sampling. Finally triangle decimation eliminates many key pomts that 
represent the swept surface geometry. 

Another limitation lies in the algorithm itself Since the output is a triangle file, 
it is very difficult to take it back to CAD systems. Despite past efforts to load triangle 
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data into CAD systems, it is usually not an easy task because of the complex geometry 
of the swept volumes. 

Because of certain limitations of tracking technologies and computing power, 
sophisticated design modification has proven difBcult to perform in visualization 
5 systems. However, information obtained from the visualization systems needs to be 
sent back to the CAD syst^. This also ^plies to generated swept volumes. To 
realize this goal, an overview 238 of a method for genmting swept volumes directly 
in the CAD system is shown in FIGURE 42. In this figure, on the right side of the 
dashed line is the impHdt modeling method shown in FIGURE 41. 

10 After the virtual part trajectory path is obtained, the trajectory is sent to the 

CAD system. This is a relatively ^mple task since the trajectory just consists of 
transformation matrices representing the position and orioitation of the instances. 
The same virtual parts are repeatedly put together according to the obtained 
tngectory, then the geometry is moged together into a single part using a Boolean 

IS operation. The resulting geometry is the swept volume. In one embodiment, the 
automatic assembly and merging are done by developing a Reality usirig the 
PFoDevetop^ program, which is the user's development API supplied by Param^c 
Tedmology Corp., the vendor of ProEngineer^. This method can be used with aiiy 
other CAD systems if a similar API is provided. 

20 Civea the positions and orientations of the instances, all of the instances are 

assembled together. In the CAD system, if usirig the merge/cutout fimction (a fimction 
to perform a . Boolean operation for two parts) to merge two parts togetha* in 
assembly mode, one basic rule is that these two parts cannot be the same, which is an 
obvious restriction. 

25 First, a copy of the part to work on as a base part is made and it is renamed as 

another part, e.g., "partbase**. Another restriction is that these two parts are explicitly 
assembled together, i.e. the second part needs to be assembled with references to 
features on the first part. In one embodiment, the ProDevdop^ program is employed 
to provide fimctions that can assemble a part to an assembly by a transformation 

30 matrix directly. The assembly perfornied this way is called "package assembly". This is 
a non-parametric move so the merge fimction can not be used if the part is assembled 
this way (This is a genml requirement for assembly modeling). The reason is also 
dear: v^ien the surfece intersections of the two parts are computed and all of the 
surfitces have to be parameterized in the same coordinate system. Also if the base part 

35 is modified, the relative position of the part relative to the base part may be changed. 
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Therefore, when assmbling the part on to the base pait (actually a copy of the 
same part), some kind of reference on the base part is created for assembling the part 
in ref^^ce to the created features. For example, some datum planes on the base part 
can be created and used with align/mate constraints to explicitly assemble the part. 
5 However, since the position and orientation of the part relative to the base part is 
known, the most natural way to do it is by creating coordinate systems on the base 
part as a datum feature and use the coordinate system constraint method to assemble 
the part. 

In a feature-based CAD modeling system, the "chunks'* of solid material from 

10 which the models are constructed are called "features". Features generally fill into 
one of the following categories: base feature, sketched feature, referenced feature or 
datum feature. The coordinate system feature for the invention can employ a 
referenced datum feature. But the situation is that it is not always practical to create 
the coordinate systems interactively since there are perhaps more than one hundred 

15 instances to deal with. 

The invention provides for creating coordinate systems automatically. Since a 
coordinate system is a complete feature, a UDF method in the CAD system can be 
used to create it. The term UDF (User Defined Feature) refers to a file that contains 
feature information. A UDF acts like a feature made up of elements. A UDF can be 

20 defined interactively in an active ses^on of the CAD system Eadi UDF consists of 
the selected features, aD their associated dimoisions, any relations between the 
selected features and a list of references for pladng the UDF on a part. Once the UDF 
is defined, it can be put into a UDF library and used to create the same types of 
features. To ilhistrate the concepts and procedures of UDF, detailed procedures of 

25 creation of the coordinate systems as UDFs are described as following. 

Every part has it's own de&ult coordinate system. Interactively, a coordinate 
system is created referring to the defiult coordinate system Actually the defiult 
coordinate system itself can be a UDF which refers to nothing. The defiult coordinate 
system is picked, named "DEFAULTCS" and saved in the UDF library. When 

30 creating a coordinate system referring to a de&ult coordinate system, the offiet values 
are spedfied along X, Y, Z directions and rotation angles about X, Y, Z directions of 
the new coordinate system relative to the defiult coordinate system The rotation 
angles are order sensitive. Also, the values provided are not important because the 
values will be modified when the UDF is used. Once the new coordinate system is 

35 created, the new created coordinate system is picked and defined as a UDF. The CAD 
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system will then go through certain procedures to define the detailed issues of the 
UDF. The two most important questions are: 
1. Which feature does the UDF refer to? 
Here, the reference is DEFAULTCS. 
5 2: What are the values that need to define the relationship of this UDF with the 

reference? 

Obviously, the X, Y, Z of&ets and 5^ Y, Z rotation angles. 
Once this UDF is defined, it is saved as PARTCS in the UDF libraiy. A 
flowchart 240 is shown in FIGURE 43 for a UDF method that employs automatic 
10 assonbly and swept volume generation. On the base part, a de&ult coordinate system 
is first created by using DEFAULTCS, then the coordinate systems fi-om PARTCS 
UDF is created by referendng this DEFAULTCS. The actual positions and 
orientations are dedded by the values we obtained fi-om the trajectory calculation. 
Also, a DEFAULTCS is created on the part that is going to be assembled. The part 
15 can then be placed by refermong the PARTCS on the base part and DEFAULTCS on 
the part. Once they are assembled, the merge fiuiction is used to merge the part into 
the base part. All the instances can be processed tlus way and finally the base part 
represents the parametric model of the swept vohmie. The complicated processes of 
sur&ce intersecting, mergirig and re-parameterization are taken care of in^de the 
20 CAD system. 

No matter what kind of method is used to compute the swept volumes, 
analytical or numerical or the method discussed above, the first task is to obtain the 
trajectory of the part as it moves in the space. 

When the user moves the part with his/her hands in the 3D virtual space of the 

25 VAE system, the invention determines the trajectory of the part during the motion. 
The volume of the part occupied in a certain time is called an instance. Actually, it is 
the combination of the part geometry, part position, and part ori^itation. The whole 
swept volume is the union of the all the instances. The user is given a choice whether 
he/she wants to create the swept volume while the part is moving held in his/her right 

30 hand. All the actions below will be effective if the user chooses to CTeate the swept 
volume. Once the system is in the volume creation mode, the defined volume begms 
whenever the user grabs the part and stops whenever the user releases the part. 

Referring to the scene graph in FIGURE 9, in every fi^e, the invention 
calculates the transformation matrix of the part in the global DCS using Equations 

35 4.1.1, 4.1.2 and 4.L3. 
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\partLocati(mJ^orm}=\part_ma(ra]x[pal^ (4.1.1) 
(geometry of part in global) - (geometry of part) x \partLocationJ^drm] (4. 1.2) 
(geometry of base in global) "geometry of hastt)x[baseLocationJfform] (4. 1.3) 

5 

where IpartLocationJ^orm] - transformation &am part DCS to the global 

DCS 

[part matrix] - transformation from part DCS to pabn DCS 
{palmmatrix] -transformation from palm DCS to hand DCS 
10 [hand_matrix] - transformation from the hand DCS to global DCS 

{baseLocationXform] - transformation from base part DCS to the 

global DCS 

In theory, where the swept volume is created makes iu> difiference since the 
swept volume will form a whole boundary and can be moved to anywhere. If no 
15 enviroimient evaluation is performed, the swept volume is created in the base part 
DCS. Equation 4.2 is used to calculate the transformation of the part in base part 
DCS. 

IpcailnBaseLoccUionXfcfrm^lpartjna^ 
xlBaseLoccaionJffdrm]'^ (4.2) 
20 A matrix array T is dedared that can hold the transformation matrices for 

every instance. Also, the instance number is stored before the user stops the swept 
volume. For every instance, the part geometry is copied and transformed using 
\partInBaselA)catioriXform ]. So if the base part moves, all the instances will move 
with it. The reason for the need to copy the geometry of the part is because the 
25 instances are picked individually and ind^endently. Otherwise, the instances can be 
displayed by referring to the same geometry, but they can not be pidced separately. 

The trajectory represented by T is time independent smce the trajectory totally 
depends on the transformation matrices. This is a very use&l property is discussed in 
greater detail below. 

30 Swept Volume gaieration is usually not a real time process. Sometimes, it 

may be time consuming. One question is what happens if the user is not satisfied with 
a swept volume after it is created? ( e.g. maybe the user wants to take a different 
tr^ectory path.) The invention provides for real time swept volume editing 
fonctionality before the swept volume is (seated from all the instances. The editing 

35 foncti nality indudes removal and modification. 
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If the user does not care about the infonnation or the shape of the swept 
volume between two instances, the in-between instances can be removed. The 
removal of one or more instances may change the swept volume greatly. 

In order to let the user pick the instances by his/her virtual fingers, the finger 

5 positions are computed relative to the swept vohime and the invention is aware v/hssi 
the swept volume is moving with the base. The calculation of the portions or the 
fingers in the global space is relatively mmple when the virtual hand model is fiiUy 
dexterous. For sinq)lidty, the position and orientation of the finger tip in the virtual 
hand DCS is represented by \flngerlTiHandXform\ 

10 Equations 4.3.1 and 4.3.2 are employed to bring the fingers and the swept 

volume to the global DCS so that they can be compared in the same coordinate 
system. In one embodiment, the invention anploys a built in intersection dieck 
mechanism in the gr^hical environment fecility to create some line segments on each 
finger tip on the user's right hand and call the intersection fimction to perform the 

15 mtersection of the line segments with the geometry that was copied for the instances. 
This symbolizes the finger picking. 

\fingertipXform\ = \flngerInH<mdXform\>([handjnaa^^ 

(4.3.1) 

IftngertipInBaseXfoTm] = [flngertipXfi^MBaseLocaiionXfarm] 

20 (4.3.2) 

Once the instance is picked, its geometry is removed fi-om the scene grq)h. 
Also the matrices array T is updated. A flowchart 242 of the removal mode process is 
shown in HGURE 44. The symbol "Nol" means the total number of instances. 

Beades instance removal, the invention also provides for instance modification 
25 fimctionality. This allows the user to change the shape of the swept vohraie by 
changing the position and orientation of the instances. It is kind of a "refining 
process". In many cases, the user may not want to move the instance m a large step, 
so the invention lets the instance translate and rotate in its own coordinate system 
The invention makes a transformation matrix called [modifyJfform]. All the 
30 translation and rotation in its own coordinate system are concatenated to this matrix. 
Suppose the transform matrix before the modification is [locationXform\ ( in global 
DCS or in base DCS ), then Equation 4.4 is used to get the new matrix. 

[newLocatianXform\ = [modifyJfform] x [locationXfarm] (4.4) 
The [newLocationXfofTrt] is copied into the trajectory array T. In order to 
35 assist the modification process, we use the higMighting feature to clearly indicate the 
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picked instances. In addition, three line segmrats are created t represent a physical 
coordinate system and will be displayed on the instance when the user picks an 
instance. The user can easily see vAiere the translation and rotation is going to be 
performed. 

5 In some cases, translation and rotation may be not convenient if the user wants 

to move some instances freely. It is easier sometimes to position and orient an 
instance directly by one's hands. It may not be practical to grab the instance and move 
it around since all the instances are close to each other and it is difficult to grab a 
certain instance. 

10 However, the invention can stili use a virtual finger tip to pick an instance. 

After the instance is picked, this instance is attached to the right hand DCS. Whrathe 
instance is finally placed in a certain position, global transformation matrix for the 
mstance is calculated. The new matrix can be computed by equations 4.5.1 and 4.5.2. 
In the equations, [instanceInHcmd\ represents the transformation between the instance 

15 and the hand at the time when the instance is picked. [msUmceMatrix] is the global 
matrix of the instance before it is picked. [insUmcelnBase] is the new matrix of the 
instance in base DCS. Also notice that [kandjnatrix] in the two equations are 
obtained at different time. 

20 [instancelnHandl = [imtanceMcarix]x[handjmari^^ (4.5. 1) 

[insiancelnBase] = [imtanceInHaanI\y[haruijnatn^^^ (4.5.2) 

One interesting problem is how to tell the system to stop the movement of the 
instance. Within the virtual environment, the primary interaction actor is the user's 

25 right hand since the left hand is holding a base part and the invention lets the fingers 
cany out this task. Because all of the fingers are dexterous and one finger can be used 
to pick the instance, the invention can use the distance between some other fingers to 
indicate the command. When the instance is picked, two fingers are moved dose to 
each other, i.e., let the distance between the two fingertips be smaller than some 

30 predefined value. The distance between two fingers is calculated while the instance is 
picked and moved around. If the user opens those two fingers, i.e., the distance is 
greater than a certain value, the movement of the instance is stopped and the new 
matrix is calculated using equations 4.5.1 and 4.5.2. 

This interaction method provides an additional interaction method in the 

3 5 virtual environmem. Usually 3D GUI picking is not very efficient when the user needs 
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to send a quick command. And in some cases, both the user's hands may be busy. 
Using the fingertip po^on testing can be used to generate some simple yes/no, 
start/stop commands and the implementation of the interaction is easy. 

A flowchart 246 of the instance modification process is shown in FIGURE 45. 
5 Using the instance removal, and modification, a swept volume is created. In this way, 
the evaluation is almost done before the swept volume is created. 

After the swept volume is created, the invention can load it back into the VAE 
system in the position where it is seated. For convenience, the transformation matrix 
for the first instance to represent the transformation of the swept volume is stored. 

10 The created swept volume behaves as a new part in the assembly model. The user can 
now perform the swept volume related evaluation tasks in the virtual environment 

One interesting combination is the creation of a swept vohmie while the part is 
undergoing the constrained motion. Another byproduct of the swept volume editing 
is the real time part design. Since the invention can modify the sht^e before the 

15 volume is oreated, it can be used as a design tool. Very sophisticated parts can be 
created out of very mmple geometry. 

Additionally, the created swept volume can be a reference to the subsequent 
assembly operations. As discussed above, the invention enables a complex assembly to 
be studied. For some critical parts, its path is reserved by the representation of the 

20 swept volume, which means that when other parts are assembled, they should not 
interfere with the swept volume. For example, if the assembly of an engine is bdng 
studied, it is well known that the spark plugs are the parts that need to be replaced 
sometime and it is important to make sure that their trajectory path remains dear. In 
this case, a user could sweep a spark plug in the assembly, edit it till the required path 

25 is known, then create the swept volume and load it back. The spark plug swept 
vohune would be left on the assembly v^en p^orming the assembling process for 
other parts. If no parts cut the spark plug's swept volume at their final portion, then 
the user would know for sure that the mamtainability of the engine for this item is 
guaranteed. 

30 Definitdy the collision detection plays an important role in the invention. The 

interference check is done accurately using the geometry data instead of just visually. 
The collision detection makes it possible that every operation will be valid. Real time 
collision detection is been included in the Invention. The combination usage of swept 
vohune and collision d^ection is also a powerfiil feature. For example, if a part is 

35 swept along certain paths and checked for colliaon betweoi the part and other parts, 
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and if collision occurs, the user can clearly find the portions or locations f the 
interference. 

Parametric Design Modification 

In one embodiment, the invention employs a parametric CAD system 
5 (Pro/Engineer"^ and a developer's toolkit that provides access to its database: 
ProDevelop'^(orProToolkit™). Hereinafter, ProH and ProD, respectively. ProDis 
a programming inter&ce to ProE that allows developers to directly access to the 
database of ProE to perfonn unique and specialized engineering analysis, drive 
automated manu&cturing, and mtegrate proprietary applications with ProE. 
10 Int«^u:tively, when the user wants to modify the design models, he/she just 

selects the "Modify" button and the dimensions of the selected feature show up. The 
user needs to pick the dimensions he/she wants to modify, enter new values, and then 
ask the system to "Regenerate" the part. The model is updated according to the 
modified values. This becomes difficult if the task needs to be performed non- 
15 interactively. The invention enters the database of ProiE through ProDevdop^, finds 
the dimensions of the sdected part, changes the part values to the dimensions that the 
user wants to modify, sends the changed values to the CAD system and lets the 
syst^ regenerate the part. A flowchart 246 of a process fi^r modifying dimensions of 
a part is shown in FIGURE 46. This figure shows a logical overview for deagn 
20 changes of a part within ProE. 

In the virtual assembly environment, the user can pick a part, recognize the 
part and assemble the part. When the user wants to perform some deagn 
modification to a part, the invention starts the ProD application program, tells the 
ProD application the part we want modify, asks ProD to go into ProE database to find 
25 the part, extracts the dimensions, s^s the dimenaons to virtual environment, does 
the changes in the virtual environment, sends the changed dimensions back to ProD, 
asks ProD to update the part and reloads the part into the virtual environment, as 
shown in a flowchart 248 in FIGURE 47. In this figure, the VAE system and ProE 
operate separately during the deagn modification process. 
30 The first problem of this method is that the virtual syston can hang during the 

defflgn process since it will take several minutes to just start a ProE sesdoiL Also, it 
will take some time to search the ProE database to find the part and extract the 
dimensions. Therefore, to accelerate the process, the time to start ProD should be 
eliminated and the time for searching the database should be reduced. To accomplish 
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these goals, the ProD application process should be running in parallel with the 
invention. 

When two processes are running in parallel, it is natural to iise signal handling 
methods and fimctions to set up communications between the two processes. In 
FIGURE 48, a sdiematic overview 250 is shown for parallel operation of PidD and 
the VAE system. In this figure, the dashed arrows mean signal sending. This parallel 
method is much better than the non-parallel method since a bi-directional connection 
is established. However, it also has some problems. First, there is too much signal 
handling. Please note that ProE is also running in parallel with ProD and there are lots 
of signal commimications between them. Second, although signal handling will not 
slow down the VAE system, it is not clear where the invention is when a signal comes 
from ProD. In this situation, it will be difficult to determine when and how to continue 
the design process. Thirdly, it is not easy to use the same ProD session for several 
different VAE sessions (starting a ProE session needs several minutes). Therefore, this 
integration architecture can be improved in several ways. 

One improvement is to reduce the signal handling between the VAE system 
and the ProD. If several sessions of the VAE system use the same session of ProD, 
the invention can know the ProD process and it is not necessaiy for ProD to know the 
different VAE sessions. Secondly, once the "Design Mode** in VADE is selected, the 
status of the information processing and supply from ProD is checked before anything 
else is executed. This requires that the VAE system knows the status information 
directly in ProD all of the time, not just when a signal arrives. However, ProD also 
needs to know the data processing in the VAE system A data sharing requirement 
between the two processes leads to a method of using shared memory provided by the 
opiating systons. 

The status flags of the two VAE and ProD systems are placed into shared 
memory, which is initialized by ProD. The data structure of the shared memoiy is 
defined as: 

struct CommunicationData{ 

int ProD_PID; /♦holding the pid of ProD*/ 

int requestDimension; /*ask ProD to get dimendons*/ 
int dimensionReady; /*tell VADE dimension ready*/ 
int dimensionChanged; /♦tell ProD dimension modified*/ 



int partRegenerated; 
char *partName; 



/*tell VADE part regenerated */ 
/•holding the part name */ }; 
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In the data structure, ProD_PID holds the process id of ProD. Since the 
shared memoiy is initialized by a ProD process, this value can be obtained by a simple 
system call. One reason to set a FID is that design modification is just one feature of 
the VAE system and it can communicate with ProD when a user wants to perform 
5 design modifications. At that tune, the VAE system sends a ^gnal to ProD and starts 
the communication. Here, one signal fi-om the VAE system to ProD is needed. In 
FIGURE 49, a sdiematic overview is shown for the VAE system and ProD using 
shared memory. This figure illustrates a parallel method of design modification in the 
VAE system through the CAD systm using a shared memory that employs one agoal 

10 between the VAE system and ProD. 

The pseudo-code of "checking, setting, processing" on the VAE system side is 
illustrated in FIGURE 50 and the pseudo-code of "checking, setting, processing" on 
ProD side is shown in FIGURE 5 1 . Please note that the "requestDimension" flag is set 
when the user picks a part and indicates he/she wants to modify the part. 

IS In the pseudo-code, the flags are set up and checked in different processes. 

"requestI>im^ons" and "dim^onChanged" are set in the VAE system and 
checked in ProD, \i^e "dimensionReady" and "partRegenerated** are set in ProD and 
checked in the VAE system. This procedure makes sure that the flags are harmonized 
and the VAE system and ProD know the status of each other all of the time. 

20 Usually, even a simple part has many dimensions. In the VAE sytem, in most 

cases, the user only wants to modify several key dimensions. So a dunension 
"filtering" mechanism is set up. In the part design session, the dimensions are 
identified that the users are allowed to change and they are named uang a predefined 
convention. The method used to name the dimensions that are going to be modified 

25 start with "vade_". For instance, 'Sradejength", "vade_diameter", "vade_outerdia", 
etc. When looping through the dimensions, the names of the dimensions are checked 
first and a dimension is extracted if its name follows this convention. 

The interaction between the user and the graphics system is through a "3D 
Gui", a 3D graphical user inter&ce library. The Gui can display buttons, tiles, and 

30 messages, and also can handle the selections of the buttons. In the VAE system, the 
"3D Gui" displays the dimensions and the user selects the dimensions. However, the 
user can input modified values fi'om the keyboard because entering floating numbers 
firom a "3D Gui" is not always practical. 

Although the mechanism discussed above is &st enough to perform design 

35 modifications, there are still cases where the user wants to go bade to stages before 
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the deagn changes if he/she is not satisfied with the modification results. So an "und 
change" mechanism is provided. Before a newly modified part is loaded into the VAE 
system, the old part is backed up. If the user wants to 'Wo change", the invmition 
switches back to the old saved part. If the user wants to keep the new part, then the 

5 old part is deleted. . 

The achievement of fast design modifications in virtual environmrat througih 
the CAD system can greatiy enhance the viability and fimctionality of the applications 
of virtual reality technology in design and manufecturing. It proves that virtual reality 
can go beyond the fimctionality of just being a visualization tool and serve an 

10 important role in the integration and extension of CAD systems. 
Finger Twirling 

The model for the virtual hand can be enhanced for more accurate 
prformance and simplified incorporation of haptic feedback. For example, the 
simulated skin of the virtual hand can be improved as the numbw of saisors around 

15 the fingers oftheCYBERCH-OVE are increased. Instead of checking for the release 
status of a gripped part, every firae, the gripped part is checked fi)r gripping 
conditions again so that an improperiy gripped part would be dropped. Also, a part 
can be twirled if it is gripped by the same two fingers as in the two previous fiMes. 
Basic mechanical principles are applied to determine the amount of twiri based on 

20 finger movement. 

The virtual skin on the fingers of the virtual hand are simulated through 
sensors which are line segments attached to the fingers. For each fi^ae, the endpoints 
of these line segments are used for intersection traversal. Twenty four line spgments 
are used fi>r each finger in three drcles of nine, equispaced. Five sensors are set up on 

25 the pahn to enable the gripping of parts between the fingers and the pahn. The two 
point gripping method is used to decide the gripping status of the parts. Since the 
number of sensors has increased, the skill level for a user is reduced which prevents 
the parts to be gripped unrealistically. To prevoit parts from bdng grabbed on the 
rear side of the hand, the sensor pmr is checked if it forms a feasible combination for 

30 fiiir gripping before evaluating the two point gripping method. 

The gripping status of a grabbed part is diecked every firame. In the other 
model discussed above, a part once gripped would is made a child object of the hand. 
This resulted in the part foUowing the wrist translational and rotational motions. The 
part would not be available fiar intersection traversal as it moved from the global DCS 
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and was made the child of the palm DCS. This prevented the gripping status of the 
gripped part to be checked every frame. 

Twirling involves the manipulations of a part usually using mainly finger 
movements. This functionality is important to define a hand model as dexterous. 
5 Twirling is accomplished in two steps. First, a part is grabbed and in the second step 
it is twirled by the finger movem^tts. The gripping status of a part is recorded and 
checked by the InteractionManago' discussed above and the fimctions of the hand 
class are called when the part is twiried. 

In FIGURE 52, a flow chart 200 illustrates the twirl process for the hand 

10 model. Moving fix)m the start block, the logic advances to a block 202 where sensor 
data is retrieved firom a CYBERGLOVE and a FLOCK OF BIRDS virtual reality 
device. The log^c flows to a decision block 204 where a determination as to Aether 
an intersection with a part is detected. If fidse, the logic moves to a block 220 where 
the scene is updated in the VAE system and then the logic steps to an end block and 

15 terminates. However, if the determination at the decision block 204 is true, the logic 
advances to a decision block 206. A determination is made as whether the user is 
attempting to grip the part. If &lse, the logic moves to the block 220 and repeats 
substantially the same actions discussed above. But, if the determination is true at the 
block 206, the logic moves to a block 208 where the part is grabbed by the virtual 

20 fingers of the virtual hand. 

At the dedsion block 210, a determination is made as to whether the 
CYBERGLOVE sensors are gripping the part If fidse, the logic moves to the block 
220 and repeats substantially the same logic discussed above. Else, the logic moves to 
a block 212 where the current transform of the gripped part is determined. The logic 

25 advances to a block 214 where the tvnri transform is calculated based on the finger 
movements of the user of the CYBERGLOVE. Next, the logic steps to a block 216 
where the twirl matrix is premultiplied by the part matrix. The logic advances to a 
block 218 where the current sensor data from the CYBERCHX)VE is stored as 
previous sensor data. The logic flows to a block 220 and updates the scene in the 

30 VAE system. Lastly, the logic moves to the end block and terminates. 

Additionally, in FIGURE 53, a scene graph 183 of the dynamic coordinate 
systems (DCS) for twiriing a virtual part with virtual fingers in the VAE system is 
illustrated, which is similar to the other scene graphs discussed above. However, in 
this case, the part DCS 178 is under a finger DCS, which is directly under the pahn 
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DCS 182. Also, the pahn DCS 182 is under the hand DCS 184 is directly under the 
global DCS 186. 

FIGUKE 54 illustrates a schematic overview 222 of finger locations on a part 
for twirling. Initially, a first finger gripping point and a second finger gripping point 
5 are disposed at Al and Bl, respectively, on a part 223. After twirling, the new 
gripping points of the first finger and the second finger are A2 and B2, respectively. 
The angle between the initial gripping points and the second gripping points is 
represented by 0. 

Consider the two gripping points ^, and -5, in fi^e "n." In fi^me "ni- 1**, 
10 the two gripping points occupy the positions and respectively. The axis of 
rotation of the part passes through the origin in the direction of 0 . This axis 0 can 
be calculated using the following relation 

B,-S,^{a,-A,)+0x{S,-A,) (5.1) 
The three components of rotation are given by the following equations 

BA. 

Assuming there is no slip, 

(i,-ii,)x^ = 0 (5.4) 
We obtain 0^ 

- iR,xBAy]-iRxBA,] 
BAx +BAy +BAt 

Where, 

R = (B,-B,)-iA,-A,) (5.6) 
BA = ifi,-A,) (5.7) 
The angle of rotation § of the part is given by the relation 
25 \^ = ^J^f+&f+0f (5.8) 

The translation of the part f is approximated to the average of the difference 
of the portions of points a and 6 as the fingers gripping the object move in opposite 
direction approximately by the same distance 

f = &zA]lidlZA) (5.9) 
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All the above calculations are done in the part's co-ordinate system. The 
translation and rotation matrices are premultiplied to the current part matrix in the 
Global space. 

FIGURE 55 illustrates a system for a di^t 10 comprising components of a 
computer suitable for executing an application program embodying the present 
inventioa In FIGURE 55, a processor 12 is coupled bi-directionally to a memory 14 
that encompasses read only memory (ROM) and random access memory (RAM). 
ROM is typically used for storing processor specific machine code necessary to 
bootup the computer comprising client 10, to enable input and output fimctions, and 
to carry out other basic aspects of its operation. Prior to running any application 
program, the machine language code comprising the program is loaded into RAM 
within memory 14 and then executed by processor 12. Processor 12 is coupled to a 
display 16 on which the visualization of the HTML response discussed above is 
presented to a user. Often, programs and data are retained in a nonvolatile memory 
media that may be accessed by a compact disk-read only m^ory (CD-ROl^ drive, 
compact disk-read/write memory (CD-RAV) drive, optical drive, digital versatile disc 
(DVD) drive, hard drive, tape drive and floppy disk drive, all generally indicated by 
reference numeral 18 in FIGURE 55. A network mter&ce 22 couples the processor 
12 to a wide area network such as the Internet. 

As noted above, the invention can be distributed for use on the computer 
system for the client 10 as machine instructions stored on a memory media such as a 
floppy disk 24 that is read by the floppy disk drive. The program would then typicalfy 
be stored on the hard drive so that when the user elects to execute the application 
program to carry out the present invention, the machine instructions can readily be 
loaded into memory 14. Control of the computer and selection of options and input 
of data are implemented using input devices 20, which typically comprise a keyboard 
and a pomting device such as a mouse (ndther separately shown). Further details of 
system for the client 10 and of the computer comprising it are not illustrated, since 
they are generally well knovm to those of ordinary skill in the art. 

As described in detail above, the invention presents a complete scenario for 
assembly design. Multiple parts can be manipulated efficiently for assembly 
evaluations. Constrained motion simulation and dynamic simulation assist the 
assembly evaluation operation. The overall process is simulated realistically mimicking 
the physical assembly processes. Dynamic behaviors of objects in the virtual 
environment are implemented using physical laws and increases realistic feeling. 
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Interactive editing of assembly path and swept volume directly by the user is 
achieved in the virtual environment. The editing includes swept instance addition, 
removal, and modifications of positions and orientations. The editing of the swept 
volume before the assembly geometry is finalized ensures the validity and significance 
5 of the swept volume. The swept volume is also converted to a parametric model and 
loaded back into the CAD system for fiuther evaluation. Collision detection 
fimctioiudity is also provided in the VAE system. 

Bi-directional interaction is achieved between the VAE and CAD systems. 
For relatively simple parts, the interaction t^cle is real-time. For sophisticated parts 
10 with many dimensions, the interaction speed may be slower. However, with more 
powerful computers, real time interaction could be achieved with even the most 
complex parts. 

Test cases have been carried out with models fi'om industry. Results from the 
invention compare very well with results fi'om the Boothroyd methodology (which is 
15 widely used in industry) for predicting assembly time. 

A significant deviation fiom reality occurs in the process of gripping the part 
This occurs primarily fiom the "sluggishness" of VR systems created by trackiiig 
fi^uency, tracking latency, firame rates, and graplucs latency. This sluggishness does 
not seem to a£fea gross motor movements (moving a part into place and aligning it) 
20 except in acute situations with large data bases. However, it significantly affects fine 
motor movements (e.g. finger and wrist movements). 

While the preferred embodiment of the invention has been illustrated and 
described, it will be appredated that various changes can be made therein without 
departing fi^om the spirit and scope of the invention. 
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The embodiments of the invention in which an exclusive property or privilege 
is claimed are defined as follows: 

1. A method for providing a virtual environment for simulating the 
arranging of a plurality of parts into an assembly, comprising: 

(a) creating a model in a design environment for each part, each model 
having a geometry that corresponds to a part; 

(b) translating each model into a virtual part in the virtual environment, the 
design environment bdng integrated with the virtual environment; 

(c) enabling each virtual part to be positioned in the virtual environment, 
wherein the pomtioning of each virtual part enables a simulation to be performed for 
the arranging of the plurality of parts into the assembly; and 

(d) enabling the simulation to be modified, a modification enabling another 
simulation to be performed, and when the modification causes a change in the virtual 
part, causing the corresponding model to automatically include the change to the 
virtual part. 
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Ab 
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FIG. 20 



I 



CONSTRAINT 
NEEDS TO BE 
APPLIED 



188 



Fiist Constraint 

? 



Yes 



Second Constraint 



NO 


Redundant? 


► 





Snapping 



No 



Special 
Cases? 



No 



Special 
methods 



Yes 




Redundant? 




Yes 


1 





Yes 



No 




Q END J 
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194 



190 



Inheritance, 




FIG. 22 




FIG. 23 



^ipiiedUst 



when grabbed 
► 



196 



AppliedList 



Part 


/ ^196 




Un^pliedList 


_ when grabbqtf' 






released or placea 



if applied 



197 



u^en grabbed 




I if redund aflt 
RedundantList r 



Constraint 
Manager 



released or placed 
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Fig. 25 
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Fig. 26 
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When float in space 



196 



Appliedlist 



rr: 

y If applied 



UnappliedList 



Whengrabbec^ 
Displayer 



RedundantUst 



redundant 



197 



Fig. 29 



232 



crank property file 








LENGTH UNITS: Inch 








M7VSS UNITS: Pound 








Surface Finish: 0.001000 






Defaiilt Linear Tolerances: +- 0. 


100000 +- 0.010000 +- 0.001000 


Default Angular Tolerances: 


ANGf- 0.500000 




volume: 12.6798973 






Surface Area: 69.5717665 






Density: 1.0000000 






Mass: 12.6798973 








CENTER OF GRK/ITY W£t DefaultCSCO coordinate frame: X Y Z 




0.0000000 


1.1668295 


0.5907307 


INERTIA wrt DefaultCSCO coordinate frame: 




INERTIA TENSOR: 








Ixx Ixy Ixz 


69.0504703 


-0.0536521 


0.0000182 


lyx lyy Izz 


-0.0536521 


17.8369050 


-3.1957782 


Izx Izy Izz 


0.0000182 


-3.1957782 


68.8163268 


INERTIA at CENTER OF 


GRAVITY wpt 


(Axis aligned) 


DefaultCSCO coordinate 


frame: 








INERTIA TENSOR: 








Ixx Ixy Ixz 


47.3620913 


-0.0536521 


0.0000182 


lyx lyy Izz 


-0.0536521 


13.4120927 


5.5442472 


Izx Izy Izz 


0.0000182 


5.5442472 


51.5527601 


PRINCIPAL MOMENTS OF 


INERTIA (wrt 


MC) : 




11 12 13 


12.6224281 


47.3621611 


52.3423549 


ROTATION MATRIX from 


DefaultCSCO 


orientation to 


PRINCIPAL AXES: 




0.0015290 


-0.9999977 


-0.0015153 




0.9900096 


0.0013001 


0.1409941 




-0.1409918 


-0.0017157 


0.9900093 



Fig. 30 
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Lifiiiig Capacity for infinequent movement and short distanRRS 
(pounds) 

Men Women 


1. Both hands in front 
Or one hand at side 


60 


40 


2. Both hands 20 inches 
in front 


20 


13.3 


3. Each hand at side 


30 


20 


3. One hand or each 
5 inch out 


35 


23.3 




Fig. 32 




Fig. 33 
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Fig. 36 
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BasePait 




Create aUDF 
DEFAULTCS 



240 



Create a UDF 
DEFAULTCS 




Instance 
Positions. Angles 




Create a UDF 
PARTCS 



Assembly by 
Coordinate System 



Merge Part onto 
Base Part 



Fig. 43 



Continue 




Pick an instance 
find the index i 



I 



Remove it from 
the scene graph 



I 



Update T 



1 




N 



Remove 
more? 




242 



Fig. 44 
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Pick an instance 



Highlight the instance 
Draw local XYZ axis 



Translate, Rotate 



Cumulate the 
matrix 



N 



Free Move 



I 



Compute Matrix 




End 

modifying 

? 



I 
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Find the Part 



Extract the 
Dimensions 





Regenerate the 


< 


Part 



Modify Some 
Dimensions 



Send Modified 
Values Back 
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► 


Pick a Part to 




Start ProD 




Modiiy 


► 


Program 



Load the New 
part 



VAE 



Find the Part 



Extract the 
Dimensions 



Modify Some 
Dimensions 



Regenerate the 
Part 



ProE 



Send Modified 
Values Back 



Vig. 47 



Pick apart to 
Modify 



Start Pn>D 
Program 



250^ 



Find the Part 
and Extract 
Dimensions 



Modify 
Dimoisions 




Regennatethe 
Part 



Load the New 
part 



ProE 
Database 



Fig, 48 
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i 



Start VAE 

~~r~ 



Obtain Shared 
Memoiy 



252 




paitName 
requestDimension 
dimensionReatfy 
dimensionChanged 
PaitRegeneratcd 

Fig. 49 



i 



Start ProD 
Progcam 



I 



Initialize Shared 
Memoiy 



PickaPartto 




Modify 






If ( dimensionReady) 
{ 

display dimensions; 

let user pick dimensions; 

modify the picked dimensions; 

dimensionCfaanged = ON; /*sct the flag V 

dimensionReady - OFF; /*rcset the flag*/ 

} 

If ( partRegenerated) 

{ 

load the new part; 
r^lace the old part; 

partRegeratcd =OFF; /♦reset the flag*/ 

} 



Fig. 50 
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If ( requestDimensions) 

{ 

find the part; 

load part into memory; 

eTctract the dimensions; 

dimensionReady = ON; /*set the flag */ 
requestDimensions = OFF; /»reset the flag*/ 



If ( dimensionChanged) 

{ 

get new dimensions; 
set the new dimensions; 
regenerate the part; 

partRegenerated = ON; Z+set the flag */ 
dimensionChanged - OFF; /*reset the flag*/ 



Vig. 51 
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Q START ^ 



1 




GET BIRDS AND 
GLOVE DATA 



202 



206 




204 



208 



210 



212 



GET CURRENT 
XFORMOFPART 



'NO- 



214 



CALCULATE TWIRL 
XFORM BASED ON 
ITNGER MOVEMENTS 



216 



PREMULTJPLY TWIRL 
MATRIX TO PART 
MATRIX 



218 



STORE CURRENT 
SENSOR DATA AS 
PREVIOUS SENSOR DATA 



220 



UPDATE SCENE 
INVAE 



END ^ 



I 

200 
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